skipping duplicate Attribute mapping (same name and nameFormat)

[Andy Bennett]

Hi,

I have an SP configured such that all of the targeted-id and
persistent-id decoders are enabled in attribute-map.xml.

When our app first sees a login for a particular SP it decides whether
to use persistent-id or targeted-id as the key for our "shadow user
account". It looks for persistent-id first and then targeted-id.

Some IDPs send multivalued persistent-ids and this is causing us
problems that we'd like to fix.



We see three types of IDP who send more that one "id":

+ sends different targeted-id and persistent-id

For this we just use persistent-id and all is well.


+ sends single valued targeted-id and bi-valued persistent-id, all the same

When we receive a multi-valued persistent-id we split it into the
values, assert them equal then pick one so this case works fine.


+ sends single values targeted-id and tri-valued persistent-id, last
persistent-id matches targeted-id, first and second persistent-
ids match each other but don't match targeted-id

For this one we are in a pickle because we try to choose
persistent-id and our multi-value algorithm fails. We then have to go
and set this IDP to key by targeted-id so that it doesn't trip over on
the "all values in multi-valued persistent-ids much match" requirement.



Now, there are three types of thing which get decoded to "persistent-id":

+ urn:mace:dir:attribute-def:eduPersonTargetedID
+ urn:oid:1.3.6.1.4.1.5923.1.1.1.10
+ urn:oasis:names:tc:SAML:2.0:nameid-format:persistent

... and one type of thing which gets decoded to "targeted-id":

+ urn:mace:dir:attribute-def:eduPersonTargetedID


As the targeted-id decoder appears before the persistent-id decoder we
see "skipping duplicate Attribute mapping (same name and nameFormat)" in
the logs for the urn:mace:dir:attribute-def:eduPersonTargetedID
persistent-id decoder.




In order to try to get a better idea of who is sending what, I'd like to
change the persistent-id decoders to decode to, say
"persistent-id-{oasis,oid,mace}" then my keying algorithm could change from

"pick persistent-id then targeted-id" to "pick oasis, then oid then
mace" and I could dispense with targeted-id entirely.


If I'd been more careful when I set things up then this would be
trivial. However, I already have a bunch of IDP entries in my app which
think that they're keyed by "persistent-id" and I can't find enough
information in the logs to know which decoder was used and therefore I
can't change this to "oasis", "oid" or "mace".


Is there a way to allow attribute-map.xml entries to work where they
have the same "name" and "nameFormat" entries as a another decoder but a
different "id"? That way I can continue to use the keys I currently use
but also gradually capture the {oasis,oid,mace} information.





Thanks for your help.



Regards,
@ndy

[Cantor, Scott]

Post by Andy Bennett
I have an SP configured such that all of the targeted-id and
persistent-id decoders are enabled in attribute-map.xml.

Don't do that. You seem to be enabling both the "support the broken
format" rule and the "decode broken format into the expected form for
migration" rule. Pick one (or just flat ignore the broken form and don't
support it). You should absolutely not be using both rules, they're
mutually exclusive by intent. And it's impossible by design to have two
rules that are for the same underlying SAML data.

Post by Andy Bennett
For this one we are in a pickle because we try to choose
persistent-id and our multi-value algorithm fails. We then have to go
and set this IDP to key by targeted-id so that it doesn't trip over on
the "all values in multi-valued persistent-ids much match" requirement.

Stop supporting targeted ID, that would be my suggestion. Enable the
special decoder we provided to turn that into the proper syntax that
should be used in all cases, and use that as the required form that you're
keying off. That was the point of adding it.

Post by Andy Bennett
+ urn:oid:1.3.6.1.4.1.5923.1.1.1.10
+ urn:oasis:names:tc:SAML:2.0:nameid-format:persistent

Those are just alternate SAML transports for the same thing, and the
former is essentially unneeded unless SAML 1.1 support is required. In no
case should they ever be different anyway.

Post by Andy Bennett
In order to try to get a better idea of who is sending what, I'd like to
change the persistent-id decoders to decode to, say
"persistent-id-{oasis,oid,mace}" then my keying algorithm could change from
"pick persistent-id then targeted-id" to "pick oasis, then oid then
mace" and I could dispense with targeted-id entirely.

You don't need to split out the various rules to do that, the whole point
is to collapse them all into one and indeed dispense with targeted-id.

Post by Andy Bennett
If I'd been more careful when I set things up then this would be
trivial. However, I already have a bunch of IDP entries in my app which
think that they're keyed by "persistent-id" and I can't find enough
information in the logs to know which decoder was used and therefore I
can't change this to "oasis", "oid" or "mace".

You shouldn't have to, that's a meaningless distinction.

You're not recognizing that the decoding and serialization of the broken
"targeted-id" form is what makes it look "different". It's not different
in any underlying sense, the value part is the same, it's just that it's
scoped with a domain instead of with the the entityID qualifiers.

The special decoder plugin turns the broken scoped form into a serialized
form that looks like the "proper" encodings already serialize into so that
the application doesn't have to care.

Post by Andy Bennett
Is there a way to allow attribute-map.xml entries to work where they
have the same "name" and "nameFormat" entries as a another decoder but a
different "id"? That way I can continue to use the keys I currently use
but also gradually capture the {oasis,oid,mace} information.

No, how could it? The same name and nameFormat is by definition the same
thing on the wire. There is no possible way for it to be "different".
You'd be mapping the same data from the same element to two different
things (and you can do that already with the aliases option).

-- Scott

[Cantor, Scott]

Post by Cantor, Scott
Stop supporting targeted ID, that would be my suggestion. Enable the
special decoder we provided to turn that into the proper syntax that
should be used in all cases, and use that as the required form that you're
keying off. That was the point of adding it.

This, BTW, is of course meant to happen as part of a one-time migration in
your application data so that it lines up, as noted on the
eduPersonTargetedID page in the wiki explaining the mess we accidentally
created.

Basically, you take any scoped values you might have stored, chop the
domain, turn the value into whatever serialization you choose to use in
the regular rules (e.g. value!IdP!SP or whatever) and then you're ready to
turn off the original targeted-id support and use the newer Decoder in its
place. That will handle any IdPs that send you the broken form and hide
that from the application.

-- Scott

[Andy Bennett]

Hi,

Post by Cantor, Scott
Stop supporting targeted ID,

...this is one of the things that I'm trying to do.

Post by Cantor, Scott
...that would be my suggestion. Enable the
special decoder we provided to turn that into the proper syntax that
should be used in all cases, and use that as the required form that you're
keying off. That was the point of adding it.

I can change <base-64>@scope keys in our shadow accounts to <base-64>
and then say "use persistent-id" rather than "use targeted-id" for that
IDP. However, it doesn't work in general because some IDPs send
differently salted base-64 in targeted-id and persistent-id (see below).
Therefore I need a way to select a specific attribute per IDP as ending
up with a multi-valued persistent-id with non equivalent values, in an
arbitrary and undefined order is not useful.

Post by Cantor, Scott
You're not recognizing that the decoding and serialization of the broken
"targeted-id" form is what makes it look "different".

That's not what I meant by "different".

I see targeted-id values of <base64>@scope. Fair enough.

I see persistent-id values of SP!IDP!<base-64> and SP!IDP!<base-64>@scope.

Given a single response from a single IDP which contains multi-valued
persistent-ids and targeted-ids I see different base 64 values in
different values as some IDPs have more than one salt active for
different encoders on their side.

In the "sends single values targeted-id and tri-valued persistent-id,
last persistent-id matches targeted-id, first and second persistent-
ids match each other but don't match targeted-id" case in my original
mail, I see

targeted-id = "***@scope"
persistent-id = "SP!IDP!Y;SP!IDP!Y;SP!***@scope"

I also see things like

targeted-id = "***@scope"
persistent-id = "SP!IDP!Z;SP!IDP!***@scope"

and

targeted-id = "***@scope"
persistent-id = "SP!IDP!W;SP!IDP!W"



In transaction.log I see attribute decode lines such as

persistent-id (1 value)
persistent-id (2 values)
targeted-id (1 value)

...sometimes all in a single transaction.


I've tried talking to IDPs who send more than one base-64 value but when
we worked through the problem they couldn't choose one because they had
some SPs that relied on one and some SPs that relied on the other, the
didn't know who required and they had issues with various services when
they switched one or other off.


...so I have a need to distinguish the values sent inside attributes
with different URIs and a need to choose a particular URI on a per IDP
basis.

We are trying to be "liberal in what we accept" and we have very little
influence over what we are sent due to operational issues on the IDP side.


We're in an inter-federated environment so it's only going to become
more important to be able to deal with these issues on the SP side.





Regards,
@ndy

[Cantor, Scott]

Post by Andy Bennett
Therefore I need a way to select a specific attribute per IDP as ending
up with a multi-valued persistent-id with non equivalent values, in an
arbitrary and undefined order is not useful.

Well, one step forward, I suppose, is to make sure you never map anything
in the broken form into the alias "persisent-id". That should at least
isolate the cases to some degree. You're in a scenario where moving to
"correct" state is impossible, so you definitely need to turn off the
features that are meant to assist with migrating to "correct" state.

Post by Andy Bennett
In the "sends single values targeted-id and tri-valued persistent-id,
last persistent-id matches targeted-id, first and second persistent-
ids match each other but don't match targeted-id" case in my original
mail, I see

But you shouldn't be seeing a scoped value in that last value, the
NameIDFromScoped decoder should be stripping it. You shouldn't have
anything mapping a scoped value of any kind into that alias. Nothing I
provide does that.

Post by Andy Bennett
I also see things like

Same thing, I don't see how that's possible. The token $Name in the
formatter in the rule is eplicitly only the left hand side of the scoped
expression.

Post by Andy Bennett
and
persistent-id = "SP!IDP!W;SP!IDP!W"

And I don't see any way that's possible given the above. I think you need
to actually tell me what mapping rules you're using here. And it's
possible the underlying XML being sent is also wrong in some unforeseen
way that's causing problems with the code. Again, eliminating the
NameIDFromScoped case should clean up some of this.

Post by Andy Bennett
...so I have a need to distinguish the values sent inside attributes
with different URIs and a need to choose a particular URI on a per IDP
basis.

It's not a feature currently. Other than by explicitly handling all the
XML of course.

Post by Andy Bennett
We are trying to be "liberal in what we accept" and we have very little
influence over what we are sent due to operational issues on the IDP side.
We're in an inter-federated environment so it's only going to become
more important to be able to deal with these issues on the SP side.

You can make some small improvements by eliminating options like SAML 1.1
support, requiring a persistent NameID, that kind of thing. But that
mostly just eliminates duplication that likely is not hurting anything
other than looking ugly. What you can't do is map attributes differently
based on the IdP, not without creating per-IdP resource URLs. The mapping
layer by itself cannot do it.

-- Scott

[Andy Bennett]

Hi,

Post by Cantor, Scott

Post by Andy Bennett
Therefore I need a way to select a specific attribute per IDP as ending
up with a multi-valued persistent-id with non equivalent values, in an
arbitrary and undefined order is not useful.

Well, one step forward, I suppose, is to make sure you never map anything
in the broken form into the alias "persisent-id". That should at least
isolate the cases to some degree. You're in a scenario where moving to
"correct" state is impossible, so you definitely need to turn off the
features that are meant to assist with migrating to "correct" state.

Agreed.

Post by Cantor, Scott
And I don't see any way that's possible given the above. I think you need
to actually tell me what mapping rules you're using here.

Here is the appropriate excerpt from my attribute-map.xml

-----
<!-- A persistent id attribute that supports personalized anonymous
access.
-->

<!-- First, the deprecated/incorrect version, decoded as a scoped
string:
-->
<Attribute name="urn:mace:dir:attribute-def:eduPersonTargetedID"
id="targeted-id">
<AttributeDecoder xsi:type="ScopedAttributeDecoder"/>
<!-- <AttributeDecoder xsi:type="NameIDFromScopedAttributeDecoder"
formatter="$NameQualifier!$SPNameQualifier!$Name"
defaultQualifiers="true"/>
-->
</Attribute>

<!-- Second, an alternate decoder that will decode the incorrect form
into the newer form.
-->
<Attribute name="urn:mace:dir:attribute-def:eduPersonTargetedID"
id="persistent-id">
<AttributeDecoder xsi:type="NameIDFromScopedAttributeDecoder"
formatter="$NameQualifier!$SPNameQualifier!$Name"
defaultQualifiers="true"/>
</Attribute>

<!-- Third, the new version (note the OID-style name): -->
<Attribute name="urn:oid:1.3.6.1.4.1.5923.1.1.1.10"
id="persistent-id">
<AttributeDecoder xsi:type="NameIDAttributeDecoder"
formatter="$NameQualifier!$SPNameQualifier!$Name"
defaultQualifiers="true"/>
</Attribute>

<!-- Fourth, the SAML 2.0 NameID Format: -->
<Attribute name="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent"
id="persistent-id">
<AttributeDecoder xsi:type="NameIDAttributeDecoder"
formatter="$NameQualifier!$SPNameQualifier!$Name"
defaultQualifiers="true"/>
</Attribute>
-----


AIUI, the "First" declaration (for targeted-id) causes the "Second" (for
persistent-id) to be ignored in all cases.

Is that correct?

Post by Cantor, Scott
And it's
possible the underlying XML being sent is also wrong in some unforeseen
way that's causing problems with the code. Again, eliminating the
NameIDFromScoped case should clean up some of this.

My theory is that the XML contains strange values but I can't confirm
that with the logs I have.

What would I need to change in the rules above?

Post by Cantor, Scott
You can make some small improvements by eliminating options like SAML 1.1
support, requiring a persistent NameID, that kind of thing. But that
mostly just eliminates duplication that likely is not hurting anything
other than looking ugly.

Rather than drop support entirely (because we do speak to some non SAML
2.0 IDPs) I'd rather decode those NameIDs to different webserver variables.

Post by Cantor, Scott
What you can't do is map attributes differently
based on the IdP, not without creating per-IdP resource URLs. The mapping
layer by itself cannot do it.

I don't really want to do that in the SP. I'd rather just have the SP
decode everything separately and then I can pick the format I like the
most in the app. In the app it's easy for us to do things on a per IDP
basis.

The trouble with moving to separate decodings is that I don't know which
inputs are mapping to which persistent-id values for the current IDPs
I've seen.





Regards,
@ndy

[Cantor, Scott]

Post by Andy Bennett
Here is the appropriate excerpt from my attribute-map.xml
-----
<!-- Second, an alternate decoder that will decode the incorrect form
into the newer form.
-->
<Attribute name="urn:mace:dir:attribute-def:eduPersonTargetedID"
id="persistent-id">
<AttributeDecoder xsi:type="NameIDFromScopedAttributeDecoder"
formatter="$NameQualifier!$SPNameQualifier!$Name"
defaultQualifiers="true"/>
</Attribute>

That one you need to disable because you've left the earlier one enabled.
Hopefully it's ignored, but that's relying on unspecified behavior. That's
why it's commented out by default, because the earlier one is enabled. I
don't think it should be hurting anything, but you don't want to depend on
that.

The others are what I'd expect.

Post by Andy Bennett
AIUI, the "First" declaration (for targeted-id) causes the "Second" (for
persistent-id) to be ignored in all cases.
Is that correct?

Maybe. Duplication of rules is unsupported. If the code changed in some
way and started taking the last one that it saw, that would be within the
documented behavior. It's unspecified how duplicate rules are handled.

Post by Andy Bennett
My theory is that the XML contains strange values but I can't confirm
that with the logs I have.

Given what you posted, there probably have to be values that are
questionable, if not wrong. There shouldn't be any scoped values in any of
the data mapped to "persistent-id", because that would mean the scoped
values are actually inside a NameID element. That's not exactly illegal in
that the values of a persistent NameID are just opaque, but it suggests
configurations that are very odd and certainly not a good idea.

Post by Andy Bennett
What would I need to change in the rules above?

Just commenting out the duplicate rule for safety sake, but the rest of
what you're seeing is just bad data and people sending nonsense, or the
same data in multiple ways. They're all IdP mistakes.

Post by Andy Bennett
Rather than drop support entirely (because we do speak to some non SAML
2.0 IDPs) I'd rather decode those NameIDs to different webserver variables.

You can't decode the same thing on the wire to different variables based
on the IdP. Not without application overrides, per IdP resources, and
per-application extraction rules. There is no support in the extractor for
rules based on IdPs.

Based on your rules, any cases where there are values in "persistent-id"
that don't match each other is a mistake on the part of the IdP that I
couldn't explain without seeing what they're sending and their
configurations, and then it would just be a matter of pointing out what
they screwed up. There's no way to fix it once it's been sent. And if they
have apps depending on bad data, there is no way out but to fix things in
some deliberate way.

-- Scott