Re: SSL client connection error - SSL_Handshake(): Peer not recognized or badly formatted message received.

[Scott Klement <midrange-l@xxxxxxxxxxxxxxxx>]

Paul,

This question would be better answered by Brad Stone as he uses the SSL 
APIs regularly.  I prefer the GSKit ones, myself...

But, I assume that GSKit and SSL APIs work the same, here, just have a 
different syntax.  (That's pretty much the norm with these...)

The stuff you set in the system values are the only protocols that are 
allowed, the system won't let the APIs override these settings.  But, 
the APIs can further restrict the choices.

So, for example:

Sysval allows TLS 1.1 and 1.2
API specifies TLS 1.0, TLS 1.1 and TLS 1.2
Result:  Only TLS 1.1 or 1.2 will work.  If client doesn't support 
these, you get 'Peer not recognized'

Sysval allows SSL 2 and 3, TLS 1.0,1.1 and 1.2
API specifies TLS 1.2
Result: Only TLS 1.2 will work, if client doesn't support it, 'Peer not 
recognized'

Sysval allows TLS 1.0, 1.1 and 1.2
API specifies: SSL 3
Result: Nothing will work, always get 'Peer not recognized'


The error 'Peer not recognized' can occur for other reasons besides 
these protocol mismatches.  For example, if you try to connect with an 
SSL client to a plaintext server (one that doesn't support SSL) then it 
will also produce this same error.  This is something I do all the time 
by accident :-)

The error just means that the data it received doesn't fit any SSL/TLS 
settings that you have enabled.  So if it gets a plaintext message, that 
would look like an 'unrecognized' SSL message. But also, if it's in a 
format it's not configured for, it also wouldn't recognize it.

-SK

On 6/26/2015 2:21 PM, Steinmetz, Paul wrote:
> Scott,
>
> We are using the SSL APIs.
>
> Do these SSL APIs use the 3 system value settings, QSSLCSL, QSSLCSLCTL, QSSLPCL?
> Or
> Do they have their own settings?
> Or
> Are they hardcoded?
>
> What needs to be done to fix this?
>
> Paul
>
>
>
> -----Original Message-----
> From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Scott Klement
> Sent: Friday, June 26, 2015 11:11 AM
> To: Midrange Systems Technical Discussion
> Subject: Re: SSL client connection error - SSL_Handshake(): Peer not recognized or badly formatted message received.
>
> Since he references SSL_Handshake(), I'm assuming that he's using the SSL APIs.
>
> But if someone does need info about how to configure the versions in GSKit, let me know, I can provide that...
>
>
>
> On 6/26/2015 9:10 AM, Bradley Stone wrote:
> > Hi Paul.
> >
> > What are you using to connect/communicate?  Can you get the return code?
> >
> > Do you know if the GSKit APIs are used or the standard SSL APIs are being
> > used for the connect?
> >
> > I ran into an issue with a customer on V7R1 that was trying to use V7R1 and
> > up and the SSL APIs weren't really doing things right, so on the SSL
> > Handshake API we had to tell it by sending it the proper code and that
> > cleared things up.
> >
> > Here's a link to an article I wrote about it.. it refers to GETURI but it
> > would also apply to any client application that uses the SSL APIs. (the
> > GSKit APIs may have a different setting).
> >
> > http://www.fieldexit.com/forum/display?threadid=170
> >
> > Brad
> > www.bvstools.com
> >
> > On Fri, Jun 26, 2015 at 8:06 AM, Steinmetz, Paul <PSteinmetz@xxxxxxxxxx>
> > wrote:
> >
> > > I'm receiving this error when trying to connect to a remote server.
> > >
> > > SSL_Handshake(): Peer not recognized or badly formatted message received.
> > >
> > > V7R1, TR10, latest CUM 15142 and all groups
> > >
> > > I've confirmed DCM has proper CA, both root and intermediate.
> > > Remote server has TLS1.0 disabled, TLS1.2 is currently being used for
> > > other connections to that server.
> > > I'm thinking this is either a SSL protocol issue or cipher issue.
> > > I know when the I is the server, the DCM application defaults need to be
> > > changed to allow TLS1.2 , TLS1.1 and disable SSL 3.0, SSL2.0
> > > Also cipher defaults need to be changed.
> > >
> > > Are there similar settings for when the I is the client?
> > > I've seen other posts with this error, but did not see the final
> > > resolution.
> > >
> > >
> > > - - - - - - - - - - - - - - - - - - - - - - - C O N N E C T I O N   F E E
> > > D B A C K -
> > > About to connect() to XXXXXX-web.XXX.net port 443 (#0)
> > >     Trying XXX.XXX.XXX.X... connected
> > > SSL_Handshake(): Peer not recognized or badly formatted message received.
> > > Closing connection #0
> > > SSL connect error
> > >    ************End of Data********************
> > >
> > >
> > > Thank You
> > > _____
> > > Paul Steinmetz
> > > IBM i Systems Administrator
> > >
> > > Pencor Services, Inc.
> > > 462 Delaware Ave
> > > Palmerton Pa 18071
> > >
> > > 610-826-9117 work
> > > 610-826-9188 fax
> > > 610-349-0913 cell
> > > 610-377-6012 home
> > >
> > > psteinmetz@xxxxxxxxxx
> > > http://www.pencor.com/
> > >
> > > --
> > > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
> > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > To subscribe, unsubscribe, or change list options,
> > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > Before posting, please take a moment to review the archives
> > > at http://archive.midrange.com/midrange-l.