Hi Community,

I've install a STIX/TAXII server and tried polling for 1 specific collection. It's currently still running and it has been more than 1 hour now.

1. Has anyone tried using a Stix/Taxii polling? How long does it usually take to finish one collection?
2. Is there a minimum specs requirement for the server? Does this affect the performance of the polling?

Thanks! :) 

Actually it depends on the data. which collection are you trying to download? It was working normal when I was using it.

Hi Mr. Eugene,

Thank you for the answer,

The poll is now finish. It took around 4.5 hours.

I used the HAILATAXII.COM site and polled the Abuse_ch collection.

Thank.

If you don't specify a begin date, it pulls all the data. If the data is big, it takes long time to complete. I used the following command and it was completed in seconds.

arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.Abuse_ch --today --output /tmp/ --debug

Instead of --today option you can use "--days 7". it pulls data from last 7 days.

Hi Mr. Eugene,

Thank you for that information, we will schedule it to run daily.

But we are having trouble with the flex now. I used the flex config file that was with the "arcsight_stix_taxii.zip" file there seems to be no problem with the flex script. the csv file says it was processed but it is not on our ESM.

I took a look at the agent.log and saw this:

Here are the files - it says that it was processed:

here is the flexconn script that I used:

delimiter=,

text.qualifier="

comments.start.with=\#

trim.tokens=true

contains.empty.tokens=true

token.count=11

token[0].name=otype

token[0].type=String

token[1].name=observable

token[1].type=String

token[2].name=indicatorType

token[2].type=String

token[3].name=firstdetecttime

token[3].type=String

token[4].name=lastdetecttime

token[4].type=String

token[5].name=score

token[5].type=String

token[6].name=confidence

token[6].type=String

token[7].name=producer

token[7].type=String

token[8].name=rdata

token[8].type=String

token[9].name=description

token[9].type=String

token[10].name=altid

token[10].type=String

event.name=__stringConstant("Collective Intelligence Feed")

event.deviceFacility=__toLowerCase(otype)

event.deviceSeverity=__toLowerCase(confidence)

event.message=observable

event.deviceCustomDate1Label=__stringConstant("First Detected Time")

event.deviceCustomDate1=__createSafeLocalTimeStamp(firstdetecttime,"yyyy-MM-dd HH:mm:ss Z")

event.deviceCustomDate2Label=__stringConstant("Last Detected Time")

event.deviceCustomDate2=__createSafeLocalTimeStamp(lastdetecttime,"yyyy-MM-dd HH:mm:ss Z")

event.requestUrl=__ifTrueThenElse(__contains(otype,"url"),observable,)

event.sourceAddress=__oneOfAddress(__ifTrueThenElse(__contains(otype,"ipv4"),observable,))

event.deviceCustomIPv6Address1=__stringToIPv6Address(__ifTrueThenElse(__contains(otype,"ipv6"),observable,))

event.sourceDnsDomain=__ifTrueThenElse(__contains(otype,"fqdn"),__toLowerCase(observable),)

event.sourceUserName=__ifTrueThenElse(__contains(otype,"email"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"md5"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"sha1"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"sha256"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"sha512"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"uuid"),observable,)

event.deviceCustomNumber1Label=__stringConstant("Score")

event.deviceCustomNumber1=__safeToRoundedLong(score)

event.deviceCustomNumber2Label=__stringConstant("asn")

event.deviceCustomNumber2=__oneOfLong(asn)

event.deviceCustomString1Label=__stringConstant("Sources")

event.deviceCustomString1=__toLowerCase(producer)

event.deviceCustomString2Label=__stringConstant("Reference")

event.deviceCustomString2=altid

event.deviceCustomString3Label=__stringConstant("Indicator Types")

event.deviceCustomString3=indicatorType

event.deviceCustomString4Label=__stringConstant("Related Data")

event.deviceCustomString4=rdata

event.deviceCustomString5Label=__stringConstant("Description")

event.deviceCustomString5=description

event.deviceProduct=__stringConstant("CIF")

event.deviceVendor=__getVendor("Threat Intel")

event.deviceVersion=__getVendor("2.1")

Thanks for the help.

First of all, check agent.out.wrapper.log and search for "First event" lines.  If you see something like "First event from [Threat Intel|CIF|..... ] received, it means the parsing is OK. Then check your destination settings on the connector and search relevant logs on the ESM.

If you don't see "Firs event" message in the logs, you need to check if there is a parsing issue.

Hi Mr. Eugene,

We used a regex flex script and it works. i dont know what the problem of the delimted flex script.

Thanks.

Hi Mr. Eugene and Community,

Have anyone experienced this error?

We are trying to poll on Hailataxii.com and the collection name is "EmergingThreats_rules".

After a few minutes -> The polling stops -> then a csv file was written on the output folder but has no entry -> then an error prompts. (see screenshot below)

This is what the log file says:

2018-12-12 10:39:12,586 : arcsight_stix_taxii : DEBUG : Error occurred while running client: list index out of range
Traceback (most recent call last):
File "c:\python27\lib\site-packages\arcsight_stix_taxii\client.py", line 837, in main
related_objects=args.related_objects)
File "c:\python27\lib\site-packages\arcsight_stix_taxii\client.py", line 206, in run_poll
csvout.write(stix_object, **csv_row_options)
File "c:\python27\lib\site-packages\arcsight_stix_taxii\storage\local.py", line 489, in write
for otype, row in self.rows(stixobject, **kwargs):
File "c:\python27\lib\site-packages\arcsight_stix_taxii\storage\local.py", line 359, in rows
type_ = str(malware_instances.types[0])
File "c:\python27\lib\site-packages\mixbox\typedlist.py", line 79, in __getitem__
return self._inner.__getitem__(key)
IndexError: list index out of range

Thank you for the support! :)

Hi Josh,

What version of the client do you use?

arcsight-taxii-client -v

Thanks,

Bart

Hi Josh,

What version of the client do you use?

arcsight-taxii-client -v

Thanks,

Bart