On an Internet website I have a database containing about 20,000 records, which will grow by about 30,000 records per year. Each record has a unique Id.
I have 2 applications that reside on about 500 PC's. I cannot join the applications.

The first application creates files, always in a known folder. The created file may contain a value equal to one of the unique Id's in the Internet database.

The second application needs to:
1. scan the known folder  - preferably at 5 second intervals. If a new file exists, extract the Unique Id from the file and see if it exists in the Website database.
2. If is does, transfer the file to the website via FTP
3. If is does not, do nothing

It can occur that the unique record on the website database is updated after the PC file has been created so when I run the scan I need to check every file on the PC against the Website.
Because the number of records & files is set to grow I am worried about performance.

Any suggestions on how I should design the process?

[Edited by TimTDP on 06-07-2008 at 07:00 AM GMT]