Aprendizaje automático de la interacción de funciones mediante redes neuronales autoadaptativas

anotación 

La predicción de la tasa de clics (CTR), cuyo objetivo es predecir la probabilidad de que un usuario haga clic en un anuncio o producto, es fundamental para muchas aplicaciones en línea, como anuncios en línea y sistemas de asesoramiento (recomendación). Este problema es muy complejo porque: 1) las funciones de entrada (por ejemplo, identificación del usuario, edad del usuario, identificación del artículo, categoría del artículo) suelen ser escasas; 2) la predicción eficaz se basa en funciones combinatorias de alto orden (también conocidas como funciones cruzadas), que requieren mucho tiempo para el procesamiento manual por parte de expertos en el dominio y no son enumerables. Por lo tanto, se han realizado esfuerzos para encontrar representaciones de baja dimensión de objetos en bruto escasos y de alta dimensión y sus combinaciones significativas. 

En este artículo, proponemos un método AutoInt eficiente y efectivo para analizar automáticamente las interacciones de objetos de alto orden de los objetos de entrada. Nuestro algoritmo propuesto es muy general y se puede aplicar a características de entrada tanto numéricas como categóricas. En particular, comparamos características tanto numéricas como categóricas en el mismo espacio de baja dimensión. Luego, se propone una red neuronal autoajustable multipropósito con conexiones residuales para modelar explícitamente las interacciones de características en un espacio de baja dimensión. Con la ayuda de diferentes capas de redes neuronales autoestresadas multipropósito, es posible simular diferentes órdenes de combinaciones de características de entrada. El modelo completo se puede aplicar de manera efectiva a datos brutos a gran escala de manera integral.Los resultados experimentales en cuatro conjuntos de datos reales muestran que nuestro enfoque propuesto no solo es superior a los enfoques de pronóstico modernos existentes, sino que también proporciona un buen poder explicativo de la red.El código está disponible en .

1. Introducción 

Predecir la probabilidad de que los usuarios hagan clic en anuncios o productos (también conocido como predicción de tasas de clics) es un tema crítico para muchas aplicaciones web, como la publicidad en línea y los sistemas de recomendación [8, 10, 15]. La efectividad del pronóstico tiene un impacto directo en los ingresos finales de los proveedores comerciales. Por su importancia, está generando un interés creciente tanto en el ámbito académico como en el comercial. 

, . . -, [8, 11, 13, 21, 32]. ,   / .    , , . CTR Criteo, 30 99,99%. (). -, [8, 11, 19, 32], . , ,   -, , . <Gender=Male, Age=10, productCategory=VideoGame> . . , [8, 26].   :  ?  , .     , ,  .

,   () [26], , [27, 28]. , ,    . ,  [8, 11, 13, 38],  . , . . ,   , -  [4]. -, , , . ,   , . 

,   - [36].    ,  , . , , , (, ). , . ,    [36]. ,   ()  , , ,   . .   , . [12], . , . 

, : 

• ; 

  
  
  
  
  

• ,   , ; 

  
  
  
  
  

• , CTR , , . 

  
  
  
  
  

. 2. .   3    . 4. . 5 .  ,   6.

2.   

: 1)   -; 2) ; 3) .  

2.1    

   -, [8-10, 15, 21, 29, 43]. , Google   Wide&Deep[8] , ,    . . . ,   . [31] -   , <, , >. Oentaryo  . [24]    . 

2.2   

,  .    () [26], [27, 28].   . ,    (FFM) [16] . GBFM [7] AFM [40] .   . 

, . , NFM [13] . , PNN [25], FNN [41], DeepCrossing [32], Wide&Deep [8]  DeepFM [11] . ,   . , , . -, Deep&Cross [38]  xDeepFM [19]   . , ,  – . -, [39, 42, 44] , , . -, HOFM [5] . HOFM ,   ( 5) . ,    . 

2.3   

: [2] [12].

[2] , [35], [30] [14, 33, 43].   . [36]  - . 

[12]  ImageNet. , y = F (x) + x, , . 

3.   

(CTR) : 

1. ( CTR) x ∈ R ⁿ u v, , n -  . , u v x. 

CTR , x  , . , x ,  . . , , [6, 8, 11, 23, 26, 32]. 

,   . 

2 ( p-).  x ∈ R ⁿ p- g (x_i1 , ..., x_ip ),     , p- , g (·) - , [26] [19, 38]. , x_i1 × x_i2- , x_i1 x_i2. 

. . ,    . , .   . 

3 ( ).  x ∈ Rⁿ , - x, . 

4. Autoint:    

 AutoInt, CTR. , ,   . 

4.1  

, . .1, x, , (. . , ) . , () . ,  , . . 

,    . . 

4.2   

, . , 

M - ,  x_i - i- . x_i -  , i- (, x₁ . 2). x_i -  , i- (, x_M  . 2). 

4.3   

, (, ). ,  , . ., 

 V_i - i,  x_i - . ,  x_i - . . , , (, «»). 2 : 

q - , i- ,  x_i - - . 

, . ,  

 v_m - m,  x_m - . 

, , .2. 

4.4   

   , . , , . , . - - [36]. 

   (Multi-head self-attentive network) [36] . , [36] [20], [37].   . 

,  «-» [22], , . m, , , m. m k  () h : 

ψ (h) (·, ·) - , m k. ,  ⟨·, ·⟩. - .

W^(h)_Query, W^(h)_Key ∈ R^{d′ × d} 5 - ,  R^d   R^d′. m h, , α^(h)_{m, k}: 

(6) m ( h), , . , , , . , , : 

 ⊕ - , H - . 

, (. . ) , . , 

 W _Res ∈ R ^{d ′ H × d} - [12], ReLU (z) = max (0, z) - . 

 e_m  e^Res_m,   . . , . 

4.5   

{e^Res_m }^M_m=1, , , , ()  . CTR , : 

w ∈ R ^{d ′ H M} - , , b - , σ (x) = 1 / (1 + e^−x) .  

4.6  

- , : 

 y_j  y_ˆj - CTR , j , N - . , : 

 logloss  . 

4.7  AutoInt 

. , 5-8, , . 

, (. . M = 4), x₁, x₂, x₃ x₄ . (, 5) , , , g (x₁, x₂), g (x₂, x₃) g ( x₃, x₄)   ,   g (·) ( 2)  ReLU (·). , x₁, e^Res₁.    , , . 

, . e^Res₁ e^Res₃, , , x₁, x₂ x₃, , e^Res₁ e^Res₃, e^Res₁ g (x₁, x₂), e^Res₃ x₃ ( ). , . , g (x₁, x₂, x₃, x₄) e ^Res ₁ e ^Res ₃, g (x₁, x₂) g (x₃ , x₄) . , . 

, ,  AutoInt  , , . , [3, 18]. 

 

, [11, 19, 32],  nd , n - , d - . : {W ^(h) _Query, W ^(h) _Key, W ^(h) _Value, W_Res}, L- L × (3dd ′ + d ′ Hd),    M. , d ′ HM + 1 . , O (Ldd′H). , H d ′ (, H = 2 d′ = 32 ), . 

 

. -,  ( )  O(Mdd' + M2d' ) . O(Mdd' + M2d') . H  (), O(MHd' (M + d)) . ,  H,d  d ' .  AutoInt  5.2. 

5.  

. : 

RQ1)   AutoInt  CTR?  ? 

RQ2)  ? 

RQ3)  ?  ? 

RQ4)  ? , . 

5.1   

5.1.1  . . 1. 

Criteo.  CTR, 45   . 26   13 .  

Avazu.  , , . 23 , / .  

KDD12.  KDDCup 2012, . CTR, , (1 > 0, 0 ), FFM [16].  

MovieLens-1M.  .  ( )  3 , , . 3    –  , 3.  

. -, ( ) «<>», {10, 5, 10}  Criteo, Avazu  KDD12 . -, , , z  log²(z) z> 2,     Criteo Competition. -, 80%    .  

5.1.2  . : 

AUC ROC (AUC)  , CTR , . AUC . 

Logloss.   logloss , 10, .  

, AUC  Logloss  0,001 CTR, [8, 11, 38]. 

5.1.3  . : A) , ; )   , ; C) , . . 

LR (). LR .

FM [26] (B). FM .

F [40] (B). AFM - , . FM, ,   . 

DeepCrossing [32] (C). DeepCrossing     .

NFM [13] (C). NFM . . 

CrossNet [38] (). -,  Deep&Cross, .

CIN [19] (C). ,  xDeepFM, .

HOFM[5] (). HOFM .    Blondel et al. [5]   [13], , . 

 CrossNet  CIN,  Deep&Cross  xDeepFM,  plain DNN (. . 5.5). 

5.1.4  .  TensorFlow[1].  AutoInt  d 16, - 1024. AutoInt  , d 32.  ( ) - . , [34] {0.1 - 0.9} MovieLens-1M, ,   . 200 - NFM, . CN CIN AutoInt. DeepCrossing  100, . ,   . ,  Adam [17] , . 

5.2  (RQ1) 

.  , 10 , 2. : 1) FM AFM, , LR , , CTR; 2)  - , ; ,  DeepCrossing  NFM , FM AFM  ,  (, CIN ); 3) HOFM FM  Criteo  MovieLens-1M, , ; 4) AutoInt  .  

 Avazu CIN ,  AutoInt  AUC,  Logloss. ,  AutoInt  ,  DeepCrossing, , , . 

  4. , LR - . FM NFM , NFM   . CIN, , - . . , AutoInt ,  DeepCrossing  NFM. 

( ) . 3, CIN  AutoInt  . 

, ,  AutoInt  . CIN, AutoInt  -. 

5.3  (RQ2)  ,   AutoInt. 

5.3.1  .  AutoInt  , , , . , , . 4, , . ,   KDD12 MovieLens-1M, , . 

5.3.2  .  ( 4). ,     , . , ( ), , . 

5. , , , , , . , . . , . , , , . 

5.3.3  .    d, . KDD12 , , . MovieLens-1M. 24, . , , , . 

5.4  (RQ3) 

, . ,  AutoInt  . MovieLens-1M.  

, , . . 7 () , .  ,  AutoInt  <Gender=Male, Age=[18-24), MovieGenre=Action&Triller> (. . ).  , , , . 

, . . . 7 (). , <Gender, Genre>, <Age, Genre>, <RequestTime, ReleaseTime> <Gender, Age, Genre> (. . ) , . 

5.5 (RQ4) 

     CTR [8, 11, 19]. , ,  AutoInt  .  AutoInt+ : 

• Wide&Deep [8]. Wide&Deep  ; 

  
  
  
  
  

• DeepFM [11]. DeepFM      ; 

  
  
  
  
  

• Deep&Cross [38]. Deep&Cross-  CrossNet  ; 

  
  
  
  
  

• xDeepFM [19]. xDeepFM - CIN . 

  
  
  
  
  

5 ( 10 ) .  : 1)   ,   , , , , , ,   AutoInt  ; 2) , AutoInt+ ,     CTR. 

6.   

CTR, -, .     , . . , . ,   , AUC  Logloss    . 

    -. ,  AutoInt  , , . 

[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, et al. 2016. TensorFlow: A System for Large-Scale Machine Learning.. In OSDI, Vol. 16. 265–283.

[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.

[3] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1798–1828.

[4] Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H Chi. 2018. Latent Cross: Making Use of Context in Recurrent Recommender Systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 46–54.

[5] Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. 2016. Higher-order factorization machines. In Advances in Neural Information Processing Systems. 3351–3359.

[6] Mathieu Blondel, Masakazu Ishihata, Akinori Fujino, and Naonori Ueda. 2016. Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms. In International Conference on Machine Learning. 850–858.

[7] Chen Cheng, Fen Xia, Tong Zhang, Irwin King, and Michael R Lyu. 2014. Gradient boosting factorization machines. In Proceedings of the 8th ACM Conference on Recommender systems. ACM, 265–272.

[8] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al.Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM, 7–10.

[9] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 191–198.

[10] Thore Graepel, Joaquin Quiñonero Candela, Thomas Borchert, and Ralf Herbrich. 2010. Web-scale Bayesian Click-through Rate Prediction for Sponsored Search Advertising in Microsoft’s Bing Search Engine. In Proceedings of the 27th International Conference on International Conference on Machine Learning. 13–20.

[11] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-machine Based Neural Network for CTR Prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, 1725–1731.

[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.

[13] Xiangnan He and Tat-Seng Chua. 2017. Neural factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. ACM, 355–364.

[14] Xiangnan He, Zhankui He, Jingkuan Song, Zhenguang Liu, Yu-Gang Jiang, and Tat-Seng Chua. 2018. NAIS: Neural attentive item similarity model for recommendation. IEEE Transactions on Knowledge and Data Engineering 30, 12 (2018), 2354–2366.

[15] Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. 2014. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising. ACM, 1–9.

[16] Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. 2016. Fieldaware factorization machines for CTR prediction. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 43–50.

[17] Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.

[18] Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. 2011. Unsupervised learning of hierarchical representations with convolutional deep belief networks. Commun. ACM 54, 10 (2011), 95–103.

[19] Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1754– 1763.

[20] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In International Conference on Learning Representations.

[21] H. Brendan McMahan, Gary Holt, D. Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, et al. 2013. Ad Click Prediction: A View from the Trenches. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1222–1230.

[22] Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-Value Memory Networks for Directly Reading Documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1400–1409.

[23] Alexander Novikov, Mikhail Trofimov, and Ivan Oseledets. 2016. Exponential machines. arXiv preprint arXiv:1605.03795 (2016).

[24] Richard J Oentaryo, Ee-Peng Lim, Jia-Wei Low, David Lo, and Michael Finegold. Predicting response in mobile advertising with hierarchical importanceaware factorization machine. In Proceedings of the 7th ACM international conference on Web search and data mining. ACM, 123–132.

[25] Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang.Product-based neural networks for user response prediction. In Data Mining (ICDM), 2016 IEEE 16th International Conference on. IEEE, 1149–1154.

[26] Steffen Rendle. 2010. Factorization machines. In Data Mining (ICDM), 2010 IEEE 10th International Conference on. IEEE, 995–1000.

[27] Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web. ACM, 811–820.

[28] Steffen Rendle, Zeno Gantner, Christoph Freudenthaler, and Lars Schmidt-Thieme. Fast context-aware recommendations with factorization machines. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval. ACM, 635–644.

[29] Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th international conference on World Wide Web. ACM, 521–530.

[30] Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 379–389.

[31] Lili Shan, Lei Lin, Chengjie Sun, and Xiaolong Wang. 2016. Predicting ad clickthrough rates via feature-based fully coupled interaction tensor factorization. Electronic Commerce Research and Applications 16 (2016), 30–42.

[32] Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and JC Mao. 2016. Deep crossing: Web-scale modeling without manually crafted combinatorial features. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 255–262.

[33] Weiping Song, Zhiping Xiao, Yifan Wang, Laurent Charlin, Ming Zhang, and Jian Tang. 2019. Session-based Social Recommendation via Dynamic Graph Attention Networks. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. ACM, 555–563.

[34] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15, 1 (2014), 1929–1958.

[35] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems. 2440–2448.

[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 6000–6010.

[37] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. In International Conference on Learning Representations.

[38] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & Cross Network for Ad Click Predictions. In Proceedings of the ADKDD’17. ACM, 12:1–12:7.

[39] Xiang Wang, Xiangnan He, Fuli Feng, Liqiang Nie, and Tat-Seng Chua. 2018. TEM: Tree-enhanced Embedding Model for Explainable Recommendation. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1543–1552.

[40] Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. Attentional factorization machines: learning the weight of feature interactions via attention networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, 3119–3125.

[41] Weinan Zhang, Tianming Du, and Jun Wang. 2016. Deep learning over multi-field categorical data. In European conference on information retrieval. Springer, 45–57.

[42] Qian Zhao, Yue Shi, and Liangjie Hong. 2017. GB-CENT: Gradient Boosted Categorical Embedding and Numerical Trees. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1311–1319.

[43] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for ClickThrough Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1059–1068.

[44] Jie Zhu, Ying Shan, JC Mao, Dong Yu, Holakou Rahmanian y Yi Zhang. 2017. Bosque de incrustación profunda: publicación basada en bosques con funciones de incrustación profunda. En Actas de la 23ª Conferencia Internacional ACM SIGKDD sobre Descubrimiento de Conocimiento y Minería de Datos. ACM, 1703-1711.