Denial of service event filtering triggered

Hi

I have several events that I process as one common event using a flex connector:

Jul 22 08:10:28 02-IBZ-KPXGTW-08 desktop-users-srv[2640834]: 100.000.0.0:33333 c0ad05273533495f VERIFY OK: depth=0, SN=First Name Last Name, GN=First Name Last Name, CN= First Name Last Name, O=Organization location, OU=organization number, title=job title, L=City, ST=Area, street=Street, 12, C=UA, serialNumber=D2860A18EHF2C8A6
Jul 22 08:10:28 02-IBZ-KPXGTW-08 desktop-users-srv[2640834]: 100.000.0.0:33333 2b1b87219c3f3e8e [First Name Last Name] Peer Connection Initiated with [AF_INET] 100.000.0.0:33333
Jul 22 08:10:28 02-IBZ-KPXGTW-08 desktop-users-srv[2640834]: 2baa95c133e13816 MULTI: new connection by client 'Откидач Юлія Ігорівна' will cause previous active sessions by this client to be dropped. Remember to use the --duplicate-cn option if you want multiple clients using the same certificate or username to concurrently connect.
Jul 22 08:10:28 02-IBZ-KPXGTW-08 desktop-users-srv[2640834]: c01a044a53b7e127 MULTI: Learn: 100.000.0.0 -> First Name Last Name /100.000.0.0:33333

For processing, I use an extra processor and multiline

my configuration file:

# CEF Parser Configuration File

replace.defaults=true
trim.tokens=true
comments.start.with=#

multiline.starts.regex=(\\w{3} \\d{2} \\d{2}:\\d{2}:\\d{2})\\s(\\S+)\\s(\\S+\\[\\d+\\]):[\\s\\S]*?CN=([^,]+).*
multiline.ends.regex=->\\s.*
regex=(\\w{3} \\d{2} \\d{2}:\\d{2}:\\d{2})\\s(\\S+)\\s(\\S+\\[\\d+\\])(.*)

token.count=4

token[0].name=Date
token[0].type=TimeStamp
token[0].format=MMM dd HH:mm:ss

token[1].name=HostName
token[1].type=String

token[2].name=ServiceId
token[2].type=String

token[3].name=Message
token[3].type=String

additionaldata.enabled=false

event.deviceVendor=__stringConstant(Test)
event.deviceProduct=__stringConstant(Test)
event.endTime=Date
event.deviceHostName=HostName
event.deviceCustomString1=ServiceId
event.message=Message

extraprocessor.count=2

extraprocessor[0].type=regex
extraprocessor[0].filename=dsns_real_extra1
extraprocessor[0].field=event.message
extraprocessor[0].flexagent=true
extraprocessor[0].clearfieldafterparsing=false

extraprocessor[1].type=regex
extraprocessor[1].filename=dsns_real_extra2
extraprocessor[1].field=event.message
extraprocessor[1].flexagent=true
extraprocessor[1].clearfieldafterparsing=false

and extra processor file:

# CEF Parser Configuration File

replace.defaults=true
trim.tokens=true
comments.start.with=#

regex=[\\s\\S]*:\\s(\\d.+):(\\d+)[\\s\\S]*?CN=([^,]+),\\sO=([^,]+),\\sOU=([^,]+),\\stitle=([^,]+),\\sL=([^,]+),\\sST=([^,]+),\\sstreet=([^,]+(?:,\\s[^,]+)?)(?:,\\sC=[^,]+)?[\\s\\S]*?Learn:\\s([^\\s]+).*

token.count=10

token[0].name=SourceAddress
token[0].type=IPAddress

token[1].name=SourcePort
token[1].type=Integer

token[2].name=CN
token[2].type=String

token[3].name=O
token[3].type=String

token[4].name=OU
token[4].type=String

token[5].name=Title
token[5].type=String

token[6].name=L
token[6].type=String

token[7].name=ST
token[7].type=String

token[8].name=Street
token[8].type=String

token[9].name=Destination
token[9].type=IPAddress

additionaldata.enabled=false

event.sourceAddress=SourceAddress
event.sourcePort=SourcePort
event.deviceCustomString2=CN
event.deviceCustomString3=O
event.deviceCustomString4=OU
event.deviceCustomString5=Title
event.deviceCustomString6=L
event.flexString1=ST
event.flexString2=Street
event.deviceAddress=Destination

The problem is that for every incoming event I get a Denial of service event filtering triggered event

As far as I know, this is a reaction to trying to give one of the fields more data than it can hold, but I don't understand in which field this is happening and I wouldn't say that there is too much information going somewhere.

They say that because of this, some data is compressed and may not arrive in full. Please tell me whether this will affect the work of collecting events and whether it is possible to lose part of the information for a certain field/fields?

Thanks in advance

Bohdan

  • 0  

    My guess would be that you're exceeding the message field after merging the events together.  The message field has a limit of 1023 and if you exceed that limit (or limits of any fields for that matter), you'll see the denial of service messages you're seeing.  The deviceCustomString fields have higher limits (4000), so it's possible you could try to use those and then clear after parsing.  

  • 0 in reply to   

    Thanks, this worked, but only for the events presented in the discussion above. However, my connector uses two extra processors because the events have two types of structures. I replaced event.message with deviceCustomString1 and enabled clearfieldafterparsing=true, but the structure, which is much larger, stopped sending data to the console, that is, I get information in the fields only from smaller events. If I leave true only for the first extra processor, and make it false for the second one, the situation does not change - the events arrive, but the information in the fields is displayed only for those that are smaller, as in the example.

  • 0 in reply to   

    I modified my config file and everything seemed to be fine, but my previous problem is still present and this time with a field that is unused. I don't understand. How this works?

    New file:

    # CEF Parser Configuration File

    replace.defaults=true
    trim.tokens=true
    comments.start.with=#

    multiline.starts.regex=(\\w{3} \\d{2} \\d{2}:\\d{2}:\\d{2})\\s(\\S+)\\s(\\S+\\[\\d+\\]):[\\s\\S]*?CN=([^,]+).*
    multiline.ends.regex=->\\s.*
    regex=(\\w{3} \\d{2} \\d{2}:\\d{2}:\\d{2})\\s(\\S+)\\s(\\S+\\[\\d+\\])[\\s\\S]*?(\\d.+):(\\d+)[\\s\\S]*?CN=([^,]+),\\sO=([^,]+),\\sOU=([^,]+),\\stitle=([^,]+),\\sL=([^,]+),\\sST=([^,]+),\\sstreet=([^,]+(?:,\\s[^,]+)?)[\\s\\S]*Learn:\\s([^\\s]+).*

    token.count=13

    token[0].name=Date
    token[0].type=TimeStamp
    token[0].format=MMM dd HH:mm:ss

    token[1].name=HostName
    token[1].type=String

    token[2].name=ServiceId
    token[2].type=String

    token[3].name=SourceAddress
    token[3].type=IPAddress

    token[4].name=SourcePort
    token[4].type=Integer

    token[5].name=CN
    token[5].type=String

    token[6].name=O
    token[6].type=String

    token[7].name=OU
    token[7].type=String

    token[8].name=Title
    token[8].type=String

    token[9].name=L
    token[9].type=String

    token[10].name=ST
    token[10].type=String

    token[11].name=Street
    token[11].type=String

    token[12].name=Destination
    token[12].type=IPAddress

    additionaldata.enabled=false

    event.deviceVendor=__stringConstant(Finally)
    event.deviceProduct=__stringConstant(Finally)
    event.endTime=Date
    event.deviceHostName=HostName
    event.deviceCustomString1=ServiceId
    event.sourceAddress=SourceAddress
    event.sourcePort=SourcePort
    event.deviceCustomString2=CN
    event.deviceCustomString3=O
    event.deviceCustomString4=OU
    event.deviceCustomString5=Title
    event.deviceCustomString6=L
    event.flexString1=ST
    event.flexString2=Street
    event.deviceAddress=Destination

    Problem in console:

    What is this rawEvent field?

  • 0   in reply to 

    If you've set preserve raw event on the connector then it would save the original event to the raw event field.  That has a limit of 4000 characters, just like the deviceCustomString fields.  If you end up with values beyond 4000, that's when you'll see the truncate messages.  Likely when the event merging is happening, it's going beyond 4000 characters which is why you're seeing these messages.  

  • 0 in reply to   

    Okay, can I disable this saving right now or is it only done during the installation and configuration of the connector?

  • 0 in reply to   

    I found that function was disable by default, but still have a problem. If it's not a problem and can't be fixed, can I just ignore it? Or I can simply increase the maximum size of the rawEvent field in agent.default.properties file. Is this a good idea and won't it lead to even bigger problems?

  • Verified Answer

    +1   in reply to 

    Increasing the size of rawEvent could end up causing problems downstream if the larger value gets passed.  ESM will also have a limit of 4000 on that, so it will have to deal will either truncating or dropping the large event values.  It would be up to you to evaluate if you feel you're missing valuable information on the truncated events, to determine how you might want to handle this.  

  • 0 in reply to   

    Although I get a denial of service notification, all the necessary data from the event comes through without issue. I don't really need to remove this notification, I just wanted to understand how it works because I get the data I need (it is structured so that 90% of the information I need is at the beginning of the event and 10% of the information is at the end of the event). I wonder how ArcSight provides the rest of the information if the event was shortened.