|
Ditto that.--
-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of
Jerry Draper
Sent: Wednesday, November 25, 2015 2:45 PM
To: Midrange Systems Technical Discussion
Subject: Re: SFTP client Expect 5.43 via PASE, maximum times allowed depending on remote OS.
Never had files that large.
I just kept watching the percentage in the piped output file and kept adjusting the timeout until it worked.
J
On 11/25/2015 11:43 AM, Steinmetz, Paul wrote:
Jerry,
Did you ever have or try files in the 100mb to 1 gb range.
This is where I'm seeing most of my issues.
Paul
-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf
Of Jerry Draper
Sent: Wednesday, November 25, 2015 2:40 PM
To: Midrange Systems Technical Discussion
Subject: Re: SFTP client Expect 5.43 via PASE, maximum times allowed depending on remote OS.
Just less than 12MB.
480 works for us.
J
On 11/24/2015 5:43 PM, Steinmetz, Paul wrote:
Jerry,
1) Thanks for the sample, reviewing..
2) When you say "very large PRICAt" file, what is the size?
3) You stated you up the timeout to 480, I never got timeout greater than 290 to work.
Paul
-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf
Of Jerry Draper
Sent: Tuesday, November 24, 2015 5:29 PM
To: Midrange Systems Technical Discussion
Subject: Re: SFTP client Expect 5.43 via PASE, maximum times allowed depending on remote OS.
For most xfers we set the timeout to 20.
In the case of the very large PRICAT file we up the timeout to 480 and then reset it back to 20 for the next in a series of gets.
We rename the file on the server to an archive folder after the xfer.
480 was determined after some trial and error.
The overall .sh script includes a byte count check of the file size on the server vs the file size received on the IFS. Any discrepancy causes a text message alert to the on call staff member.
Jerry
while {[gets $files file] != -1} {
if { $file != "ARCHIVE" && $file != "ls -1" && $file != "sftp> " } {
set timeout 480
send "get $file\n"
expect {
default {exit 2}
"not found" {exit 3}
"sftp>"
}
set timeout 20
send "rename $env(RPATH1)PRICAT/$file $env(RPATH1)PRICAT/ARCHIVE/$file\n"
expect {
default {exit 2}
"not found" {exit 3}
"sftp>"
}
send "!cp $env(LPATH1)$file $env(LPATH2)$file\n"
expect {
default {exit 2}
"not found" {exit 3}
"sftp>"
}
} else {
expect {
"sftp>"
}
}
}
close $files
On 11/24/2015 10:26 AM, Steinmetz, Paul wrote:
Jerry,
Could you please supply a sample of the code of how you modified the Expect timeout.
Paul
-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf
Of Jerry Draper
Sent: Monday, November 23, 2015 6:23 PM
To: Midrange Systems Technical Discussion
Subject: Re: SFTP client Expect 5.43 via PASE, maximum times allowed depending on remote OS.
We programmed an sFTP system using Expect because our sFTP partner will not allow key exchange.
We had to modify the Expect timeout on large files as Scott suggests below.
This could be done dynamically by checking the file size before
initiating the Expect sFTP
Jerry
On 11/23/2015 3:13 PM, Scott Klement wrote:
Paul,
1) Expect is not "one of my products". It's a public domain
program available from NIST.
There are a lot of tools available for IBM i that fall into "my
products", HTTPAPI, FTPAPI, TN5250, HSSFR4, XLPARSER4, JDBCR4, etc.
2) I do not have any other versions of it, I haven't needed them.
Feel free to get the latest sources and build your own version for
PASE if you are so inclined.
3) I do not consider any of the behavior you mentioned to be a bug
in Expect. Your script for Expect can be modified to check the
timeout in a loop, thus allowing it to take much longer. Or to
turn off the timeout completely, etc as you like.
Also, it's more useful to know the version of Expect vs. the
operating system being used. As Expect is not part of the
operating system, the version of the OS really shouldn't come into play, here.
-SK
On 11/23/2015 10:45 AM, Steinmetz, Paul wrote:
Scott,
I want to clarify that this is an Expect 5.43 issue (one of your
products), which is not supported by IBM.
Second, majority of the SFTP and SSH documentation does not apply
when Expect is used.
Third, based on my testing and attempting to change config files,
the ssh_config and sshd_config files are not used.
ClinetAliveCountMax ; ClinetAliveInterval ; TCPKeepAlive had no impact.
The reason this product was installed years ago, we needed an
SFTP solution for when the remote site could/would NOT support
the use of keys, only user and password.
Originally, only one remote vendor.
Now we are using for seven vendors.
Could you confirm my findings?
Do you have a newer version of Expect?
Is there a fix or workaround for this issue?
I've been testing sending the same file(s) via Expect 5.43 to 3
different servers using V7R1 as the client.
1) Remote Vendor OS unknown
2) Remote Local Production i5 V7R1
3) Remote Local Linux CENTOS 6.7
CALL PGM(QP2SHELL) PARM('/QOpenSys/usr/bin/sh' +
'-c' +
'/home/SFTPXFER/PAULS/paulsP05.expect > +
/home/SFTPXFER/PAULS/paulsP05.output')
It appears Expect 5.43 has a default "set timeout" of 20 seconds
when no "set timeout" is coded.
Files taking longer than 20 seconds will not complete.
By adding a "set timeout NNN" to the script, longer/larger files
can be sent.
However, the "set timeout NNN" has different maximums depending
on the remote OS.
When the "set timeout NNN" is above the maximum, file never
starts sending, connection is closed.
Sending to the Remote Vendor OS unknown, "set timeout 290"
works, higher values fail.
Sending to the Remote Local V7R1 i5, "set timeout 120" works,
higher values fail.
Sending to the Remote Local Linux CENTOS 6.7, "set timeout 119"
works, higher values fail.
Below are my testing results, showing the maximums allowed,
sending from a V7R1 OS client to:
Remote Vendor OS unknown Rem Production i5 V7R1 Rem
Local Linux CENTOS 6.7
7mb 20 sec 12 sec 12 sec
10mb 23 sec 12 sec 12 sec
15mb 23 sec 12 sec 12 sec
20mb 49 sec with ST 30 12 sec 12 sec
25mb 49 sec with ST 30 12 sec 13 sec
50mb 1m 49 sec with ST 60 15 sec 15 sec
75mb 1m 52 sec with ST 60 17 sec 17 sec
100mb 2m 49 sec with ST 90 21 sec 19 sec
125mb 2m 54 sec with ST 90 44 sec with ST 30 41
sec with ST 30
150mb 3m 46 sec with ST 120 44 sec with ST 30 45
sec with ST 30
200mb 4m 50 sec with ST 150 50 sec with ST 30
58 sec with ST 30
300mb 7m 42 sec with ST 240 60 sec with ST 30 58
sec with ST 30
400mb 8m 38 sec with ST 290 1m 49 sec with ST 60
1m 44 sec with ST 60
500mb 8m 40 sec with ST 290 1m 49 sec with ST 60
1m 46 sec with ST 60
600mb 10m with ST 290 88% 1m 57 sec with ST 60 1m
47 sec with ST 60
700mb 2m 35 sec with ST 90 2m 33 sec
with ST 90
800mb 2m 57 sec with ST 90 2m 48 sec
with ST 90
900mb 3m 31 sec with ST 120 3m 14 sec
with ST 110
1gb 3m 39 sec with ST 120 3m 24 sec with
ST 110
1.1gb 3m 51 sec with ST 120 3m 24 sec
with ST 110
1.2gb 3m 55 sec with ST 120 3m 38 sec
with ST 110
1.3gb 4m 00 sec with ST120 98% 4m 00 sec
with ST 119 88%
Paul
-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On
Behalf Of Scott Klement
Sent: Wednesday, November 18, 2015 3:00 PM
To: Midrange Systems Technical Discussion
Subject: Re: SFTP client via PASE, large files possibly timing
out
Paul,
The 'set timeout' in Expect makes perfect sense. That didn't
occur to me before seeing your recent messages, but it makes a lot of sense.
Expect is not part of SSH (like SFTP is). It is a generic
program that pretends to be a user at a terminal, and automates
what the user might do... so it keys in values, and sits and
waits for responses to be sent to the "screen" (the virtual
screen it's reading under the covers, not your actual display)
and based on what it sees on the screen, it does something in
response, etc. It can be used to automate any user interaction with a PASE command-line environment.
So if you've written your script to timeout during a put/get
command, it makes perfect sense that it's killing the file
transfer when the timeout occurs. It doesn't know what the
command is doing under the covers, it only knows what's displayed to the screen.
So, my question to you... why code expect to do a timeout during
the actual put/get command? are you having trouble with it 'freezing'
during file transfers and never completing?
Sorry this response was a bit slow... I was out of the office for
a few days.
-SK
On 11/14/2015 11:43 AM, Steinmetz, Paul wrote:
Scott,--
Those were settings recommended to check by IBM support.
By increasing "Set Timeout" from 30 to 90 included in the
.expect input script, I was successful in resolving this issue.
I tested using higher values, successful up to 296.
297 and higher fails.
Why is this?
I'd like to confirm for future for large files or when times are longer.
Paul
-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On
Behalf Of Scott Klement
Sent: Friday, November 13, 2015 12:49 AM
To: Midrange Systems Technical Discussion
Subject: Re: SFTP client via PASE, large files possibly timing
out
Paul,
Unless I'm misunderstanding your symptoms, you're looking at the
wrong settings, here...
My understanding is that your file transfer is looking good,
transferring data successfully until all of the sudden,
mysteriously, the transfer is cut off after 45 seconds.
But, you are checking things related to "inactivity".
Inactivity means that NOTHING is sent for the timeout period.
Inactivity would never cut you off in the middle of a file transfer that's progressing!
I know the timeout settings are set higher, but let's use 45
seconds as an example to explain what 'inactivity' timeouts mean...
if you had an inactivity timeout of 45 seconds that was
affecting you, this is what would happen: (1) You'd start the transfer.
(2) At some point (maybe right away, or maybe several minutes
later) the transfer would "freeze"
where no new data was transfering for some reason like a network
glitch or something. (3) 45 seconds after freezing up -- which
is to say, 45 seconds without a single byte being transferred --
the timeout would
kick in and disconnect you. If the transfer never froze up, the
timeout would never occur (even if the file took hours to send)
That's what an inactivity timeout is... it means things are stalled.
Nothing at all is transferred in the timeout period, not even a
single byte. As long as at least one byte is sent in that 45
seconds, it would not occur.
However, that's not how I understand your symptom. I understood
that it was cutting you off while the transfer was successfully going..
So unless i'm misunderstnading, what you're experiencing cannot
be an inactivity timeout. It's a timeout that occurs even on an
active session... right?
The 4 settings you mentioned, however...
1- an INACTIVITY timeout on your router.
2- Keepalive... this is completely unrelated.
3- FTP server settings -- completely unrelated, you are not using FTP.
Even if you were, it's not the server.
4- SSH is the right tool to check for settings on... but again,
this is an INACTIVITY timeout.
Unfortunately, I have no clue what would cause the issue you're seeing.
I've never seen this happen before. But, unless the session _is_
stalling for 45 seconds, this isn't an "inactivity" timeout.
It's a "you only have 45 seconds total to use this connection
before I cut you off" timeout, which is very different.
It's like saying to your teenaged daughter "you can only be on
the phone for 10 minutes per day". (Time limit) That's very
different from "as long as you say at least one word every 10
minutes you can stay on the phone, but if you don't say anything
for 10 minutes i'll cut you off"
(inactivity timeout)
On 11/12/2015 2:03 PM, Steinmetz, Paul wrote:
I discovered a new piece to this puzzle.
For this one specific remote server:
All files are successful that complete in 45 seconds or less.
All files taking longer than 45 seconds fail.
Somewhere, there must be a 45 second timer/timeout setting.
These have been checked and confirmed.
1- It could be at the firewall/router's idle timeout level. - 5
minutes.
2- Verify the TCP keepalive on the iSeries (CHGTCPA). - TCP keep
alive . . . . . . . . . 5
3- General FTP Server Inactivity Timeout Value see the
following
doc: The default inactivity timeout value is 300 seconds. The
default transfer timeout value is 420 seconds. Inactivity timeout .
. . . . . . 300
-http://www.ibm.com/support/docview.wss?uid=nas8N1011178
4- Timeout in SSH: Disconnecting Inactive SSH Sessions on the
IBM I The TMOUT and TIMEOUT PASE environment variables have a
value of
600 seconds. If there is no activity within an SSH session for
10 minutes
-http://www.ibm.com/support/docview.wss?uid=nas8N1013311
Paul
-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On
Behalf Of Scott Klement
Sent: Tuesday, November 03, 2015 4:39 PM
To: Midrange Systems Technical Discussion
Subject: Re: SFTP client via PASE, large files possibly timing
out
FWIW.. I have used sftp to send extremely large files, such as
whole server backups, without any problems at all. I did not
have to set up anything in particular, it just worked.
When you are having trouble with your larger files (imho, 110mb
isn't really very big) does it always stop at the exact same
byte position?
Or does it vary?
If it varies, I'd suggest this is probably a network issue
rather than a software issue. A flaky cable/switch/router/etc
is causing the data to get screwed up somewhere, causing the transfer to abort.
IF it's always the same, then I would look for some sort of
size limit being set up somewhere, most likely on the server-side.
Also, you could also try using scp rather than sftp and see if
that matters at all. Just as a lark...
On 11/3/2015 8:34 AM, Steinmetz, Paul wrote:
I changed the SFTP process to send a 3mb zip file instead of
the 110mb uncompressed file.
SFTP now working with a zipped 3mb file.
Large file issue for SFTP still exists, possibly 30 to 40mb limit.
Still looking to resolve this issue.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
http://archive.midrange.com/midrange-l.
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
http://archive.midrange.com/midrange-l.
--
--
Jerry Draper, Trilobyte Software Systems, since 1976 IBMi, Network, and Connectivity Specialists, LAN/WAN/VPN Representing WinTronix, Synapse, HiT, and others .....
(415) 457-3431; www.trilosoft.com
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
--
--
Jerry Draper, Trilobyte Software Systems, since 1976 IBMi, Network, and Connectivity Specialists, LAN/WAN/VPN Representing WinTronix, Synapse, HiT, and others .....
(415) 457-3431; www.trilosoft.com
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
--
--
Jerry Draper, Trilobyte Software Systems, since 1976 IBMi, Network, and Connectivity Specialists, LAN/WAN/VPN Representing WinTronix, Synapse, HiT, and others .....
(415) 457-3431; www.trilosoft.com
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
--
--
Jerry Draper, Trilobyte Software Systems, since 1976 IBMi, Network, and Connectivity Specialists, LAN/WAN/VPN Representing WinTronix, Synapse, HiT, and others .....
(415) 457-3431; www.trilosoft.com
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
As an Amazon Associate we earn from qualifying purchases.
This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].
Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.
midrange.com
