Verified Answers

If an answer to your question is correct, click on "Verify Answer" under the "More" button.  The answer will now appear with a checkmark.  Please be sure to always mark answers that resolve your issue as verified.  Your fellow Community members will appreciate it!  Learn more.

  • State Not Answered
  • Replies replies
    3
  • Subscribers subscribers
    6
  • Views views
    682
  • Users 0 members are here
  • Locked Locked
Related
Options
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Here's a little (Windows) utility to help parse SMTP Log files and diagnose delivery issues...

evanevery

I've been trying to run down a bunch of SMTP Auth, SPF, and 451 System Error issues for several months now.  The fact that the Message Tracker has been pretty well broken for all this time is not particularly helpful.  :-(  This generally requires me to plow through the SMTP log files manually to accumulate and organize the data I need...

After struggling for so long to find and extricate the needed info, it became obvious that it would simply be more efficient to create a tool to assist in this process.  I wrote similar scripts and executables (under linux) to process and parse the old GWAVA logs (inc sending alerts in real time), so I had a pretty good idea of what I needed to do and how to go about it.

Anyway, as I expect many folks know, there can be log messages from multiple concurrent threads intermingled in the SMG log files.  So, part of the task at hand it to actually isolate/parse/extract the associated log entries for an individual qualifying conversation.   ...and the string you are looking for likely requires you to back up in the log file to find the point of first contact if you want the whole conversation (inc Date, Time, and source IP).

- The little utility I have attached to this posting will search a specified SMG SMTP Log file (downloaded) for a particular string (case insensitive).  It can be an email address, part of an email address, an IP Address, a system error, or pretty much ANYTHING which might appear in the file.  It only contains a minimum amount of error checking, but it seems to get the job done nicely (at least for me)

- It will work to isolate, extricate, and dump any/all qualifying SMTP conversations which have that string somewhere in them.  It is designed to work efficiently such that if a matching string appears in an individual conversation more than once, it will not try to dump the same conversation over and over.

- Each "Qualifying" conversation will be isolated and dumped to an individual text file which has the same name as the logfile (YYYYMMDD) followed by the start time of the conversation (_HHMMSS).  This should make it fairly simple to open an individual "qualifying conversation" in a text editor to examine the entries.

- As the tool searches for qualifying conversations, it will display a brief summary of each match it finds in a running dialog (ThreadID, Date, Time, Source IP, Sender, and Recipient).  This summary can make it fairly simple to see, for example, which senders and domains are causing SMG to generate a "451 System Error"...  (As is my current task)

I have previously written other scripts and tools to generate a nightly report of all conversations processed by GWAVA the day before.  Essentially a chronological list of each conversation (with similar data as above), and then a summary listing of the number of SPF, IPREP, SURBL, or RBL a particular domain generates (to help identify those that need to be whitelisted), how many messages were being sent/received by each of our users (to help identify if any account was compromised by spammers), and few other salient statistics.  However, I'm not to that point yet with SMG as I'm way to busy trying to identify and resolve a myriad of delivery issues...  :-(

Anyway, if you find the attached tool useful, please let me know.  Suggestions are usually appreciated (even if never actually implemented).  :-)

SMGFind.zip

  • 0

    Here is an update to the above utility (SMGFind.EXE) and a new one (SMGSummary.EXE).  Both now share a common INI file (SMGTOOLS.INI) for settings and adjustments if/when key phrases in the SMTP log file are changed or added.

    While SMGFIND will isolate and dump any/all "conversations" in a log file which contain a particular phrase, SMGSUMMARY is designed to parse an entire SMTP log file, produce a brief summary of what it finds based on how the conversation terminates, and then dump a Comma Delimited file with summary information.  It will only dump conversations if it can't determine their final status.  This should help add search strings to the INI file if/when new situations/events are encountered.

    I started working to tie each conversation to its corresponding info in the SCAN log (that's why "OK" and "BLOCKED" CSV conversation entries show a "scan timestamp" at the end of the records), but a little birdie tells me the format of the scan log will soon be changing (possibly with a proper back-link to the associated conversation in the SMTP log.  Right now, there is no hard and fast link (identifier) between the SMTP and SCAN logs and you must use time stamps, sender, recipient, and IP info to identify the associated SCAN for a specific SMTP Conversation.  Additionally (unlike GWAVA), the SPAM results (SPF, RBL, IPREP, SPAM, etc) are only presented in the SCAN log and not passed back to the SMTP log.  I don't want to get too far down this path if there may be imminent changes pending for the log files (likely intended to address the Message Tracker Issues we are all seeing).

    At some point, these tools might become a little more "formal" (but always free).  However, right now, they are meeting my immediate needs.  Maybe other folks will find them useful as well...

    5584.SMGFind.zip

    0317.SMGSummary.zip

  • 0 in reply to 

    Work Continues...

    Here is an update to SMGFind (v1.3) which includes all of the functionality of the old SMGSummary utility as well.  SMGSummary is now obsolete:

    1. SMGFind will search all SMTP conversations in a specified SMG Log File based on SENDER email address, RECIPIENT email address, source IP address, and/or a user-specified STRING found anywhere in the individual "conversation".

    2. An SMTP conversation must match ***ALL*** specified search parameters in order to satisfy the search. (IOW: If you don't care about a particular search parameter - just leave it blank).

    3. All search parameters are case insensitive

    4. All search parameters may be provided as substrings.  (IE: You can just specify the domain or username for the Sender or Recipient email address.)

    5.  If you leave all Search Parameters blank, every single conversation in the log file will qualify.  (This is useful, but be careful as it can generate a lot of unwanted data - particularly if the "Dump Conversations" option is enabled!)

    6.  You may optionally generate a comma delimited "CSV Summary File".  If you request one, it will contain one record of salient info for each conversation which qualifies.  This will basically dump some the same info you see in the scrolling log window.  This includes some internal info (ThreadID, MsgID), as well as Date, Time, SourceIP, Sender, Recipient, and Completion Status (as identified by strings specified in the SMGTools.ini file).  Additionally, if the conversation identifies an SMTP Authorization failure, it will add the username that is "under attack".  ...and if the conversation results in a SCAN by SMG, it will add the exact time the scan was requested (to aid as a backlink from the associated SCAN log).

    7.  You may also choose to isolate/dump each qualifying conversation to an individual file by enabling the "Dump Conversations" option.  This will create individual "conversation files" which will be named {YYYMMDD_HHMMSS}.log in the SMGFind.exe folder. Be careful as this can generate a lot of files - especially if you don't enter any search criteria as it will dump every single conversation in the entire log file. You may want to run your query first without enabling this option to see how many hits you get before you actually dump the conversations.

    8.  The SMGTools.ini file can be modified to alter/add search strings which help identify the final disposition of conversations.  If you add a string to the list of "Status Code" you will want it to be unique to the resulting disposition. If a particular string is found ANYWHERE in a conversation it is assumed that the final disposition has been identified.  ...and once it finds a "Status Code" string match it considers that disposition final. Pay attention to the number of {Unknown}, {Quit}, or various RcvErrors in the on-screen summary as they might be hiding more specific dispositions.  You may want to "Dump" those conversations to determine if a "Status Code" update is in order...

    2553.SMGFind.zip

  • 0 in reply to 

    Version 1.31 contains a few minor bug fixes...

    SMGFind_v131.zip

Resources

Support
Documentation
Learning Services
Partner Programs
Privacy
Compliance
Help
Company
Privacy Policy
Terms of Use
Cookies Preferences
Accessibility
Anti-Slavery Statement
Support
Contact Us
Careers
Code of Business Conduct and Ethics
The opinions expressed above are the personal opinions of the authors, not of OpenText. By using this site, you accept the Terms of Use. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.), Hewlett Packard Enterprise Company, or Micro Focus. As of January 31, 2023, the Material is now offered by OpenText, a separately owned and operated company. Any reference to the HP, Hewlett Packard Enterprise/HPE, and Micro Focus marks is historical in nature and the HP, Hewlett Packard Enterprise/HPE, and Micro Focus marks are the property of their respective owners.