Hi Community,

I've install a STIX/TAXII server and tried polling for 1 specific collection. It's currently still running and it has been more than 1 hour now.

1. Has anyone tried using a Stix/Taxii polling? How long does it usually take to finish one collection?
2. Is there a minimum specs requirement for the server? Does this affect the performance of the polling?

Thanks! :) 

Actually it depends on the data. which collection are you trying to download? It was working normal when I was using it.

Hi Mr. Eugene,

Thank you for the answer,

The poll is now finish. It took around 4.5 hours.

I used the HAILATAXII.COM site and polled the Abuse_ch collection.

Thank.

If you don't specify a begin date, it pulls all the data. If the data is big, it takes long time to complete. I used the following command and it was completed in seconds.

arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.Abuse_ch --today --output /tmp/ --debug

Instead of --today option you can use "--days 7". it pulls data from last 7 days.

Hi Mr. Eugene,

Thank you for that information, we will schedule it to run daily.

But we are having trouble with the flex now. I used the flex config file that was with the "arcsight_stix_taxii.zip" file there seems to be no problem with the flex script. the csv file says it was processed but it is not on our ESM.

I took a look at the agent.log and saw this:

Here are the files - it says that it was processed:

here is the flexconn script that I used:

delimiter=,

text.qualifier="

comments.start.with=\#

trim.tokens=true

contains.empty.tokens=true

token.count=11

token[0].name=otype

token[0].type=String

token[1].name=observable

token[1].type=String

token[2].name=indicatorType

token[2].type=String

token[3].name=firstdetecttime

token[3].type=String

token[4].name=lastdetecttime

token[4].type=String

token[5].name=score

token[5].type=String

token[6].name=confidence

token[6].type=String

token[7].name=producer

token[7].type=String

token[8].name=rdata

token[8].type=String

token[9].name=description

token[9].type=String

token[10].name=altid

token[10].type=String

event.name=__stringConstant("Collective Intelligence Feed")

event.deviceFacility=__toLowerCase(otype)

event.deviceSeverity=__toLowerCase(confidence)

event.message=observable

event.deviceCustomDate1Label=__stringConstant("First Detected Time")

event.deviceCustomDate1=__createSafeLocalTimeStamp(firstdetecttime,"yyyy-MM-dd HH:mm:ss Z")

event.deviceCustomDate2Label=__stringConstant("Last Detected Time")

event.deviceCustomDate2=__createSafeLocalTimeStamp(lastdetecttime,"yyyy-MM-dd HH:mm:ss Z")

event.requestUrl=__ifTrueThenElse(__contains(otype,"url"),observable,)

event.sourceAddress=__oneOfAddress(__ifTrueThenElse(__contains(otype,"ipv4"),observable,))

event.deviceCustomIPv6Address1=__stringToIPv6Address(__ifTrueThenElse(__contains(otype,"ipv6"),observable,))

event.sourceDnsDomain=__ifTrueThenElse(__contains(otype,"fqdn"),__toLowerCase(observable),)

event.sourceUserName=__ifTrueThenElse(__contains(otype,"email"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"md5"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"sha1"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"sha256"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"sha512"),observable,)

event.fileHash=__ifTrueThenElse(__contains(otype,"uuid"),observable,)

event.deviceCustomNumber1Label=__stringConstant("Score")

event.deviceCustomNumber1=__safeToRoundedLong(score)

event.deviceCustomNumber2Label=__stringConstant("asn")

event.deviceCustomNumber2=__oneOfLong(asn)

event.deviceCustomString1Label=__stringConstant("Sources")

event.deviceCustomString1=__toLowerCase(producer)

event.deviceCustomString2Label=__stringConstant("Reference")

event.deviceCustomString2=altid

event.deviceCustomString3Label=__stringConstant("Indicator Types")

event.deviceCustomString3=indicatorType

event.deviceCustomString4Label=__stringConstant("Related Data")

event.deviceCustomString4=rdata

event.deviceCustomString5Label=__stringConstant("Description")

event.deviceCustomString5=description

event.deviceProduct=__stringConstant("CIF")

event.deviceVendor=__getVendor("Threat Intel")

event.deviceVersion=__getVendor("2.1")

Thanks for the help.

First of all, check agent.out.wrapper.log and search for "First event" lines.  If you see something like "First event from [Threat Intel|CIF|..... ] received, it means the parsing is OK. Then check your destination settings on the connector and search relevant logs on the ESM.

If you don't see "Firs event" message in the logs, you need to check if there is a parsing issue.

Hi Mr. Eugene,

We used a regex flex script and it works. i dont know what the problem of the delimted flex script.

Thanks.

Hi Mr. Eugene and Community,

Have anyone experienced this error?

We are trying to poll on Hailataxii.com and the collection name is "EmergingThreats_rules".

After a few minutes -> The polling stops -> then a csv file was written on the output folder but has no entry -> then an error prompts. (see screenshot below)

This is what the log file says:

2018-12-12 10:39:12,586 : arcsight_stix_taxii : DEBUG : Error occurred while running client: list index out of range
Traceback (most recent call last):
File "c:\python27\lib\site-packages\arcsight_stix_taxii\client.py", line 837, in main
related_objects=args.related_objects)
File "c:\python27\lib\site-packages\arcsight_stix_taxii\client.py", line 206, in run_poll
csvout.write(stix_object, **csv_row_options)
File "c:\python27\lib\site-packages\arcsight_stix_taxii\storage\local.py", line 489, in write
for otype, row in self.rows(stixobject, **kwargs):
File "c:\python27\lib\site-packages\arcsight_stix_taxii\storage\local.py", line 359, in rows
type_ = str(malware_instances.types[0])
File "c:\python27\lib\site-packages\mixbox\typedlist.py", line 79, in __getitem__
return self._inner.__getitem__(key)
IndexError: list index out of range

Thank you for the support! :)

Hi Josh,

What version of the client do you use?

arcsight-taxii-client -v

Thanks,

Bart

First, it's mr_ergene, not eugene :D

I can poll the feed using following command(limited the data for 7 days and enabled the debug option on the command):

arcsight-taxii-client hailataxii.com /taxii-discovery-service --no-https --auto --auth basic --username guest --poll guest.EmergineThreats_rules --days 7 --output /tmp/ --debug

My client version is 1.1.3

Hello mr_ergene,

Just to let you know, you are polling the collection guest.EmergineThreats_rules and Josh is polling guest.EmergingThreats_rules

Both are valid on hailataxii, but I don't know if both collections are the same.

When I poll both collections and using --days 356, I don't reveice any data can you confirm if this collection is empty, since you are using --days 7

thanks

Lol, both collections don't have any data for the last year. Probably those collections are not being maintained and have dirty data.

Ok thanks.

Acording to hailataxii.com the last update was on Fri May 25 15:18:06 2018 UTC. I checked guest.Abuse_CH and this collection is still up to date.

 The latest version of the ArcSight STIX/TAXII client (v 2.0.0) will have a fix for the error you have. 

The ArcSight STIX/TAXII version 2 will be soon available, and will also support STIX/TAXII 2.x

Hi Bart and Mr. Ergene <sorry for the typo :D>, 

I am using client v1.3, I tried to update it to the latest version using the command on the guide. I think this is the latest available version.

I tried to poll both emerging and emergine and both seems to prompt an error. I tried to poll for 7 days it finished with no error but has no entries. You have mentioned that for the last 365 days it is empty, this means that it would be pointless for me to poll the last 365 day. I'll just schedule it to run daily.

I polled the other collections and finished with no errors (Abuse_ch, CyberCrime_Tracker, MalwareDomainList_Hostlist, dshield_Blocklist)

Are there other website we call poll from with the arcsight-taxii-client other that hailataxii.com? you have mentioned that some collections are not maintained, is it reliable? what other site can you refer for us to use?

Thank you for the support guys! :)

Following is a list of some TI feeds I can recommend:

abuse.ch
alienvault OTX
MISP feeds
IBM X-force

Following is a list of some TI feeds I can recommend:

abuse.ch
alienvault OTX
MISP feeds
IBM X-force

The client is also tested with Anomali Threatstream.