Interesting...waiting for more upcoming Episode Keep it up Andreas

Interesting...waiting for more upcoming Episode Keep it up Andreas

Like your style

Interesting stuff))) I just want to make some correction for 6 point: regex is wrong, you omit an opening parenthesis. Regex should be like this:

|([^|]*)|

your right, ​, will correct it in the tail, Thanks for mentioning

Let me see if I can clarify some of these for you (from my experience):

2. you can name the override file anything you want as long as it ends with .subagent.sdkrfilereader.properties

3. you can set the vendor/product to whatever you want, it does NOT have to match the override filename

4. flexagent_syslog is what picks up the custom flexagent parsers, it already exists in the default subagentlist

5. if you only want the syslog smartconnector to pick up your custom flexagent parsers, then set customsubagentlist=flexagent_syslog and set usecustomsubagentlist=true

8. sometimes you just need to test with a very basic flexagent syslog parser [regex=(.*)] to figure out exactly what part of the syslog header the smartconnector will match and/or remove before being passed to your custom parser.

Hi Shaun and Andreas

Good clarifications below.

One small point on terminology - we should differentiate between an 'override' and a full custom flex parser. It doesnt change the gist of Shaun's last update, except...

For a custom (Syslog) flex, it is indeed true that it doesn't matter what you choose to call the stub-name - although using 'vendor' is the preferred/best practice (for when you're maintaining loads of flex's in the same framework/JVM.). And indeed, they are all enumerated as part of the 'flexagent_syslog' sub-agent type, so no explcit additons are required to the customsubagentlist.

For an 'override' on the otherhand - which is designed to augment/replace parts (or all) of an existing sub-agent - the naming has to match precisely the stock parser (which is semi-hard coded in to the framework, and sequenced against other sub-agents very specifically to avoid one gobbling another's event stream.) I know that's not what Shaun was trying to say, but the use of the term override should probably be avoided in this (flex) context.

A note on 'sequencing' of sub-agents... You have no (supported) control over the sequencing of the sub-agents - beyond enabling/disabling individual sub-agents through customsubagentlist (and its associated switch property.) If you must get eyes on how we have sequenced the long list of sub-agents (occasionally useful for Syslog or SNMP Unified), then send me a PM. Its a bit too knarly to expose here - no intent to sound patronising... But, then I am British :-)

And yes, our Syslog framework (Flex or stock-parser) is capable of consuming events with or without RFC compliant Syslog headers. If they are present and match a very broad range of (lauighably 'compliant') RFC forms, then they will be silently gobbled-up ahead of the parser ever seeing that part - leaving you with just the 'message body' to play-with. What has been gobbled up automagically wil be made avauilable in the pre-defined fields '_SYSLOGxxx' from withij your flex parser to map to something intelligent in the ARST schema.

If you want to see what the Syslog framework attempts to match in a header, browse agent.defaults.properties. Its not especially transparent, but you'll see what you need if you care enough to look. Oh, and PLEASE don't go adjusting agent.defaults.properties - its a sure-fire way to confuse anyone else who then tries to debug or maintain your connectors - Flex or otherwise. I won't therefore tell you where it lives... :-)

M.

Late to the party, but wanted to say thanks to Andreas, Shaun, Martin for the clarifications.

The notion that flexagent_syslog must be included in the customsubagentlist was not abundantly clear to me for sure.  I'm not a flexconnector guru by any stretch, and spent a bit of time agonizing over this same issue (not having that item included in my config) before coming back to research a bit more here and seeing that dragon slayed.  

In the flexconnector guide, the only mention of flexagent_syslog is in the example of what the syslog.prpoerties might look like (when doing a flex connector sanity check), which is a reasonable "clue", but IMHO it would be extremely helpful to explicitly note the need to include this in the customsubagentlist along with the flex itself - and highlight this in big bold font (for us old folk :-).

Found an interesting read in relation to the overall topic, of building your own parsers and cutomization around existing parsers, external mapping and an good overview of filenames, foldernames for flex-connectors

"FlexConnector deep dive and best practices" - from Protect 2014, see attached PDF.

I liked this table: 

seems the KB gods woke up from a "sleeping beauty" kind of sleep and started writing new KB articles.

This
How to enable debugging on a SmartConnector cought my attention as it mentions  enable debugging is different in different version. I did not veryify, and was running fine with log.global.debug=true until now, but maybe I was missing stuff. so not verified, but hopefully conserved for eternity here:

Connector version pre 8.2

log.global.debug=true
log.channel.file.property.package.com.arcsight=0
log.channel.file.property.maxsize=10MB
log.channel.file.property.maxbackupindex=20

Connector version 8.2 and later

log.global.debug=true
log.channel.file.property.loglevel=DEBUG
log.channel.file.property.package.com.arcsight=0
log.channel.file.property.maxsize=10MB
log.channel.file.property.maxbackupindex=20