Tidal to send email if listing of a directory provides a file
We have a process that will transfer files from a directory into another directory for processing. The file should then be deleted from the original directory once the process completes. However we have noticed that sometimes it does not complete successfully for whatever reason. I want to create a Tidal job that will check this directory for any files twice a day and send an email if there are any files in the directory. I am able to get the job to list out the directory, but I am unsure as to how to go about the 'Tracking' section. I want the Scan output: Abnormal String to be a value of *.csv, so anytime it sees a file with .csv it will send an email to myself.
Any assistance in this matter would be appreciated.
You could setup a job with a dependency that looks for a .csv file in that folder. So the job will only run when there is a .csv file in the folder. The job doesn't really have to 'do' anything, but I supposed you could have it delete the .csv file if it is no longer needed. Then have an event/action that will send an email when the job completes normally.
I think rreadenour addressed the simple case which is the default file dependency behavior in Tidal. So let me detail a best practice approach we use for other cases.
The purpose of Tidal is to run jobs only when they can succeed and file dependencies are being met. Make sure you set at least two file dependencies per file (Stable For; Exists) as files can still be in transit if you only use the Exist file dependency. You can just set an On Success job event if you want a notify to go to someone.
Also for every file watch, we like to include an SLA job step which is simply cmd /C echo SLA and define a job step that depends on the previous job that is watching for a file and notify if the file is not there by say 15:00. This successor job typically in the same job group as your file dependency check job will have a time window of 08:00 and 15:00 with Do not time out checked. We perform our notification from this second job. This is a best practice approach if you have to alert a BA who then may need to contact another part of your organization or an external party responsible for file delivery to resolve the issue by 3pm. The previous step that is awaiting files is still waiting but as soon as the file appears it will release or the BA would cancel if the party said they can't get the file out until tomorrow. This way you don't mix checking and running with notice activity in the same job step.
Wildcarded file dependencies have some gotchas. Review the Tidal Help topic "Using Wild Cards" if you're going to have dynamic file names appear which is most always the case.
For another more advanced case that Tidal alone didn't handle the way we wanted, we run a simple batch script you can host on your Tidal, Informatica, or any Windows batch server. The cases we use this approach is when users or a product are expected to place files in a given folder and another scheduled process is expected to remove them so that we want Tidal to watch and notify if the files are just piling up in this folder and not being processed. The script return a non-zero when there are zip files so that a Tidal error will raise because Exit Code = 0 is the normal tracking condition will be false.
@ECHO OFF REM count_zip_set_errorlevel REM REM %1 ... first variable is a UNC path to a directory to check REM %2 ... second variable is the folder to change to to check
ECHO Step 1: Temporarily unlink share drive as P: ECHO . NET USE P: /d ECHO Step 2: Temporarily link share drive as P: ECHO . NET USE P: %1 P: CD \%2
REM log causing problems when created in the data exchange directory. disabled. REM dir *.zip > count_zip_set_errorlevel.log
ECHO Step 3: Examine Zip file count ECHO . for /f "tokens=1,2,3 delims=.F " %%i in ('dir *.zip^|find "File"') do SET ERRORLEVEL=%%i echo "# zip files pending in %1\%2 are %ERRORLEVEL% exit %errorlevel%