Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Parse a Remote Folder to get newly created files



Permlink Replies: 8 - Last Post: Apr 23, 2015 8:35 PM Last Post By: Francisco Alvar... Threads: [ Previous | Next ]
Francisco Alvar...

Posts: 83
Registered: 11/10/06
Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 22, 2015 7:51 AM
My scenario:
An external application is creating a xml file every 5-10 seconds in a
file server.
I need to parse each file, just once, to create a record in a SQL table
on real time.
I tried some components (from LMD's and JAM's, see my post in
thirdpartytools.general) but they fail on notifing the name of the file
created on a remote PC, just report a folder is updated.
Since at the end of the day there are about than 10,000 files in the
folder I don't want to parse every single file, just the last (newest),
to check if it's already on the table...
Any help will be appreciated.
Best Regards
Francisco Alvarado
Peter Below

Posts: 1,227
Registered: 12/16/99
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 22, 2015 11:13 AM   in response to: Francisco Alvar... in response to: Francisco Alvar...
Francisco Alvarado wrote:

My scenario:
An external application is creating a xml file every 5-10 seconds in
a file server.
I need to parse each file, just once, to create a record in a SQL
table on real time.
I tried some components (from LMD's and JAM's, see my post in
thirdpartytools.general) but they fail on notifing the name of the
file created on a remote PC, just report a folder is updated.
Since at the end of the day there are about than 10,000 files in the
folder I don't want to parse every single file, just the last
(newest), to check if it's already on the table...
Any help will be appreciated.
Best Regards
Francisco Alvarado

I don't see a way around using a FindFirst/FindNext loop to scan the
files and check their dates. Can you move the scanned files to a
subfolder after you are done with them? That would cut down on the
number of files you have to scan on each round.

--
Peter Below (TeamB)
Francisco Alvar...

Posts: 83
Registered: 11/10/06
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 22, 2015 2:25 PM   in response to: Peter Below in response to: Peter Below
On 04/22/2015 12:13 PM, Peter Below wrote:
I don't see a way around using a FindFirst/FindNext loop to scan the
files and check their dates. Can you move the scanned files to a
subfolder after you are done with them? That would cut down on the
number of files you have to scan on each round.
Peter, I can't do that, the folder is there for many others to check logs.
Best Regards
Francisco Alvarado
Francisco Alvar...

Posts: 83
Registered: 11/10/06
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 22, 2015 2:40 PM   in response to: Peter Below in response to: Peter Below
On 04/22/2015 12:13 PM, Peter Below wrote:
I don't see a way around using a FindFirst/FindNext loop to scan the
files and check their dates. Can you move the scanned files to a
subfolder after you are done with them? That would cut down on the
number of files you have to scan on each round.
If I use FindFirst/FindNext, which will be the first file found?
Is there a way to be sure it will the most recent or the oldest?
Just wondering, because the file names will be like this:
Order_10.225.109.9_17c9c028-7e6a-48b5-9997-252cb74363ed.xml
Order_10.225.109.9_17 will not change and the rest are random characters.
Best Regards
Francisco Alvarado
Peter Below

Posts: 1,227
Registered: 12/16/99
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 23, 2015 1:38 AM   in response to: Francisco Alvar... in response to: Francisco Alvar...
Francisco Alvarado wrote:

On 04/22/2015 12:13 PM, Peter Below wrote:
I don't see a way around using a FindFirst/FindNext loop to scan the
files and check their dates. Can you move the scanned files to a
subfolder after you are done with them? That would cut down on the
number of files you have to scan on each round.
If I use FindFirst/FindNext, which will be the first file found?
Is there a way to be sure it will the most recent or the oldest?

No, there is no guaranteed sequence as far as I know. But that may
depend on the server OS, so why don't you run some experiments? Log the
filenames to a local file for several scans and compare the sequence...


--
Peter Below (TeamB)

Adem Meda

Posts: 495
Registered: 12/28/98
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 22, 2015 6:55 PM   in response to: Francisco Alvar... in response to: Francisco Alvar...
Francisco Alvarado wrote:

My scenario:
An external application is creating a xml file every 5-10 seconds in a
file server.
I need to parse each file, just once, to create a record in a SQL table
on real time.

Why not write another small app (a Windows Service) that runs on the file
server that does all this on there --and communicates with your remote app
through TCP/IP or whatever.
Francisco Alvar...

Posts: 83
Registered: 11/10/06
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 22, 2015 8:30 PM   in response to: Adem Meda in response to: Adem Meda
On 04/22/2015 7:55 PM, Adem Meda wrote:
Why not write another small app (a Windows Service) that runs on the file
server that does all this on there --and communicates with your remote app
through TCP/IP or whatever.
Thank you Adem, but I can't run anything on the server side.
I'm going to test with FindFirst/FindNext loop and see how it works at
the end of the day, when the number of files is high.
Right now I'm searching for examples...
Best Regards
Francisco Alvarado
Francisco Alvar...

Posts: 83
Registered: 11/10/06
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 22, 2015 10:13 PM   in response to: Francisco Alvar... in response to: Francisco Alvar...
On 04/22/2015 8:51 AM, Francisco Alvarado wrote:
My scenario:
An external application is creating a xml file every 5-10 seconds in a
file server...

I'm using FindFirst/FindNext loop to scan the remote folder and a
TDateTime variable to check if the file timestamp is newer. If it is
then I process it and at the end of the loop I update the variable.
I think it may reduce the number of files to be inserted in the database.
Best Regards
Francisco Alvarado
Francisco Alvar...

Posts: 83
Registered: 11/10/06
Re: Parse a Remote Folder to get newly created files
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 23, 2015 8:35 PM   in response to: Francisco Alvar... in response to: Francisco Alvar...
On 04/22/2015 8:51 AM, Francisco Alvarado wrote:
Any help will be appreciated.
I just ended using FindFirst/FindNext loop and it works fine so far...
Thank you all for your help.
Best Regards
Francisco Alvarado
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02