BEGIN:VCALENDAR
VERSION:2.0
METHOD:PUBLISH
CALSCALE:GREGORIAN
PRODID:-//WordPress - MECv7.33.0//EN
X-ORIGINAL-URL:https://dcn.nat.fau.eu/
X-WR-CALNAME:
X-WR-CALDESC:FAU DCN-AvH. Chair for Dynamics, Control, Machine Learning and Numerics -Alexander von Humboldt Professorship
X-WR-TIMEZONE:Europe/Berlin
BEGIN:VTIMEZONE
TZID:Europe/Berlin
X-LIC-LOCATION:Europe/Berlin
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:20260329T030000
RRULE:FREQ=YEARLY;BYMONTH=03;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:20261025T020000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=4SU
END:STANDARD
END:VTIMEZONE
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-PUBLISHED-TTL:PT1H
X-MS-OLK-FORCEINSPECTOROPEN:TRUE
BEGIN:VEVENT
CLASS:PUBLIC
UID:MEC-3a51104c66686fac95156c1a1d632bd4@dcn.nat.fau.eu
DTSTART;TZID=Europe/Berlin:20240502T120000
DTEND;TZID=Europe/Berlin:20240502T130000
DTSTAMP:20240430T132303Z
CREATED:20240430
LAST-MODIFIED:20240926
PRIORITY:5
SEQUENCE:1
TRANSP:OPAQUE
SUMMARY:Clustering in Pure-Attention Hardmax Transformers
DESCRIPTION:Next Thursday May 2, 2024:\nOrganized by: FAU DCN-AvH, Chair for Dynamics, Control, Machine Learning and Numerics – Alexander von Humboldt Professorship at FAU, Friedrich-Alexander-Universität Erlangen-Nürnberg (Germany)\nTitle: Clustering in Pure-Attention Hardmax Transformers\nSpeaker: Albert Alcalde\nAffiliation: PhD student at FAU DCN-AvH Chair for Dynamics, Control, Machine Learning and Numerics – Alexander von Humboldt Professorship.\nAbstract.  We study the behaviour in the infinite-depth limit of a transformer model with hardmax self-attention and normalization sublayers, by viewing it as a discrete-time dynamical system acting on a collection of points. Leveraging a simple geometric interpretation of our transformer connected with ideas of hyperplane separation, we establish convergence to a clustered equilibrium and prove that clusters are completely determined by special points called leaders. We apply our theoretical understanding to design a model based on our transformer to solve the sentiment analysis task in an interpretable way: the transformer filters out meaningless words by clustering them towards the leaders, identified with words carrying the sentiment of the text such as ‘amazing’ or ‘terrible’.\nWHEN\nThu. May 2, 2024 at 12:00H\nWHERE\nOn-site: Room 03.323\nFriedrich-Alexander-Universität Erlangen-Nürnberg\nCauerstraße 11, 91058 Erlangen\nGPS-Koord. Raum: 49.573764N, 11.030028E\n_\nSee all Seminars at FAU DCN-AvH\nDon’t miss out our last news and connect with us!\nLinkedIn | Twitter | Instagram\n
URL:https://dcn.nat.fau.eu/events/fau-dcn-avh-jr-02-may-2024/
ORGANIZER;CN=FAU DCN-AvH:MAILTO:
CATEGORIES:FAU DCN-AvH Jr. Seminar
ATTACH;FMTTYPE=image/png:https://dcn.nat.fau.eu/wp-content/uploads/FAUDCNAvHJrSeminar_aAlcalde_02may2024.png
END:VEVENT
END:VCALENDAR