Monday, December 31, 2012

Use Python to automatically get files and perform zipping transfer

Use Python to automatically get files and perform zipping transfer
The requirement:
The customers require to get the feed out files from the ftp server and send to them every day. The files on Solaris server is *.tar.gz with time stamp, however, the customer can not unzip the file on windows host. So the files must be unzipped and re-zipped to .zip file before sending to the customers.
Traditional solution:
1. Use Putty to login the file server to get the file path and name.
2. Use Winscp to connect to the server based on sftp protocol, change the directory to the file path which step 1 gets.
3. Get the files from the server to the local host via Winscp
4. Unzip the *.tar.gz and re zipped using PKZip (Need the PKZip is installed on the local host)
5. Send the files to the customers.
Improved solution:
1. Write a python program to get the files and unzip, re zip the files automatically.
2. Send the files to the customers.
The benifits:
1. The efficiency is improved significantly and the steps is very easy.
2. The quality is improved since the program is not prone to any error. 
Disadvantage:
Need python runtime environment. jython can not be support: ImportError: no module named glob
Python implementation:
1. The server can only accept sftp, so the ftplib can not be accepted.
2. Use Winscp command line to get the files
3. Use 7zip command line to unzip and zip
Actually, the python program just borrow some free modules and glue them together.

sftp.py
import os,time

year = time.strftime("%Y",time.localtime())
mon = time.strftime("%m",time.localtime())
day  = int(time.strftime("%d",time.localtime()))-1
fileserver = '/opt/appl/feedsout/nda_nfi/nda_nfi_vap/archive/AVG_VOL*'+str(year)+str(mon)+str(day)+'*.tar.gz '
command = 'winscp405.exe /console /log=winscp_log.txt /command "option batch on" "open sftp://abinitio:3m3m@gic3@172.26.66.226" "get '
#print command+fileserver+'.\\" "exit"'
os.system(command+fileserver+'.\\" "exit"')
#winscp405.exe /console /log=winscp_log.txt /command "option batch on" "open sftp://abinitio:3m3m@gic3@172.26.66.226" "get /opt/appl/feedsout/nda_nfi/nda_nfi_vap/archive/AVG_VOL*20091020*.tar.gz .\" "exit"

trans.py
import os,glob,time

##convert all *.tar.gz to *.zip under the current directory
filelist = glob.iglob('*.tar.gz')
for file in filelist:
os.system('"7za.exe" e '+file)
tmppath = file+str(time.time())
os.system('"7za.exe" x '+file[:-3]+' -O'+tmppath)
os.system('"7za.exe" a '+file[:-7]+'.zip '+'.\\'+tmppath+'\\*')
os.system('rd /S /Q '+tmppath)
os.system('del '+file[:-3])

No comments:

Post a Comment